Token and Positional Embeddings
Ids are integers. The model wants vectors. Two lookup tables sit between them, and the choice of the positional one shapes what the model can learn. Build a token-embedding lookup table that maps vocabulary ids to dense vectors. Build a learned positional-embedding lookup table indexed by position. Build a fixed sinusoidal positional embedding indexed by position with no parameters. Compose token and positional embeddings into a single input for a transformer block. Contrast learned and sinusoidal embeddings on length generalization and parameter count. The model's first contact with a token id is a row lookup in the token-embedding matrix. The matrix has one row per vocabulary id and one column per model dimension. The lookup returns a vector that the rest of the model treats as the meaning of the id. Backprop updates the rows that were used in the forward pass. Over training the geometry of those rows learns to encode similarity in directions. Token ids alone have no order. The model needs a second signal that tells it position one is different from position seventeen. The two dominant choices for that signal are a learned positional embedding (a second lookup table, one row per position) and a fixed sinusoidal positional embedding (a math formula with no parameters). The choice has consequences. A learned table is a parameter and…
Token and Positional Embeddings: Ids are integers. The model wants vectors. Two lookup tables sit between them, and the choice of the positional one shapes…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.