Positional Encoding — Sinusoidal, RoPE, ALiBi
Attention is permutation-invariant. "The cat sat on the mat" and "mat the on sat cat the" produce the same output without positional signal. Three algorithms fix it — each with a different bet on what "position" means. Scaled dot-product attention is order-blind. The attention matrix softmax(Q K^T / √d) V is computed from pairwise similarities. Shuffle the rows of X, get the rows of the output shuffled the same way. Nothing inside attention cares about position. That is not a bug in a bag-of-words model. For language, code, audio, video — anything where order carries meaning — it is fatal. The fix is to inject position into the embeddings somehow. Three eras of answers: Absolute sinusoidal (Vaswani 2017). Add sin/cos of position to the embedding. Simple, learnable-free, extrapolates poorly beyond trained lengths. RoPE — Rotary Position Embeddings (Su 2021). Rotate Q and K vectors by an angle proportional to position. Encodes relative position directly in the dot product. Dominant in 2026. ALiBi — Attention with Linear Biases (Press 2022). Skip embeddings entirely; add a per-head linear penalty to attention scores based on distance. Excellent length extrapolation. As of 2026, essentially every frontier open model uses RoPE: Llama 2/3/4, Qwen 2/3, Mistral, Mixtral, DeepSeek-V3, Kimi. A handful of long-context models use ALiBi or its modern variants. Absolute sinusoidal is historical. Sinusoidal…
Positional Encoding — Sinusoidal, RoPE, ALiBi: Attention is permutation-invariant. "The cat sat on the mat" and "mat the on sat cat the" produce the same…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.