BPE Tokenizer From Scratch
Bytes in, ids out, ids back to the same bytes. Build the tokenizer that every modern text model still starts from. Train a Byte-Pair Encoding vocabulary from a raw text corpus by repeatedly merging the most frequent adjacent symbol pair. Implement a deterministic merge table and apply it to fresh text to produce a stream of subword ids. Round-trip arbitrary UTF-8 input to ids and back without information loss. Reserve and protect special tokens ( , ) so they survive training and decoding. Reason about why a byte-level alphabet is the right floor for a general-purpose tokenizer. A language model never sees text. It sees integers. The map from a string to a list of integers and back is the tokenizer. Get this layer wrong and every loss curve in the training run is measuring the wrong thing. The dominant family of subword tokenizers for general text models is Byte-Pair Encoding. The idea is small. Start from a known alphabet. Find the adjacent symbol pair that appears most often in the training corpus. Merge it into a new symbol. Repeat until the vocabulary reaches the target size. Encoding new text reuses the same merge list in the same order. We will build the byte-level variant. The alphabet is the 256 raw bytes, not Unicode code points. That choice is what…
BPE Tokenizer From Scratch: Bytes in, ids out, ids back to the same bytes. Build the tokenizer that every modern text model still starts from.
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.