Subword Tokenization — BPE, WordPiece, Unigram, SentencePiece
Word tokenizers choke on unseen words. Character tokenizers blow up sequence length. Subword tokenizers split the difference. Every modern LLM ships on one. Your vocabulary has 50,000 words. A user types "untokenizable". Your tokenizer returns [UNK]. The model now has no signal about the word. Worse: the 90th-percentile document in your corpus has 40 rare words, which means 40 bits of dropped information per document. Subword tokenization solves this. Common words stay single tokens. Rare words decompose into meaningful pieces: untokenizable → un, token, izable. Training data covers everything because any string is ultimately a sequence of bytes. Every frontier LLM in 2026 ships on one of three algorithms (BPE, Unigram, WordPiece), wrapped in one of three libraries (tiktoken, SentencePiece, HF Tokenizers). You cannot ship a language model without picking one. BPE vs Unigram vs WordPiece, character-by-character BPE (Byte-Pair Encoding). Start with a character-level vocabulary. Count every adjacent pair. Merge the most frequent pair into a new token. Repeat until you hit the target vocabulary size. Dominant algorithm: GPT-2/3/4, Llama, Gemma, Qwen2, Mistral. Byte-level BPE. Same algorithm but over raw bytes (256 base tokens) instead of Unicode characters. Guarantees zero [UNK] tokens — any byte sequence encodes. GPT-2 uses 50,257 tokens (256 bytes + 50,000 merges + 1 special). Unigram. Start with a huge vocabulary. Assign each token a unigram…
Subword Tokenization — BPE, WordPiece, Unigram, SentencePiece: Word tokenizers choke on unseen words. Character tokenizers blow up sequence length. Subword…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.