Tokenizers: BPE, WordPiece, SentencePiece
Your LLM does not read English. It reads integers. The tokenizer decides whether those integers carry meaning or waste it. Implement BPE, WordPiece, and Unigram tokenization algorithms from scratch and compare their merge strategies. Explain how vocabulary size affects model efficiency: too small creates long sequences, too large wastes embedding parameters. Analyze tokenization artifacts across languages and code, identifying where specific tokenizers break down. Use the tiktoken and sentencepiece libraries to tokenize text and inspect the resulting token IDs. Your LLM does not read English. It does not read any language. It reads numbers. The gap between "Hello, world!" and [15496, 11, 995, 0] is the tokenizer. Every word, every space, every punctuation mark must be converted into an integer before a model can process it. This conversion is not neutral. It bakes assumptions into the model that cannot be undone later. Get this wrong and your model wastes capacity encoding common words with multiple tokens. "unfortunately" becomes four tokens instead of one. Your 128K context window just shrank by 75% for text heavy in multi-syllable words. Get it right and the same context window holds twice as much meaning. The difference between "this model handles code well" and "this model chokes on Python" often comes down to how the tokenizer was trained. Every API call you make to GPT-4…
Tokenizers: BPE, WordPiece, SentencePiece: Your LLM does not read English. It reads integers. The tokenizer decides whether those integers carry meaning or…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.