Attention Variants — Sliding Window, Sparse, Differential
Full attention is a circle. Every token sees every token, and memory pays the price. Four variants bend the shape of the circle and recover half the cost. Full attention costs O(N²) memory and O(N²) compute in sequence length. For a 128K-context Llama 3 70B that is 16 billion attention entries per layer, times 80 layers. Flash Attention (Lesson 12) hides the O(N²) activation memory but does not change the arithmetic cost — every token still attends to every other token. Three classes of variants change the topology of the attention matrix itself: Sliding window attention (SWA). Each token attends to a fixed window of neighbors, not the full prefix. Memory and compute drop to O(N · W) where W is the window. Gemma 2/3, Mistral 7B's first layers, Phi-3-Long. Sparse / block attention. Only selected pairs (i, j) get scored; the rest are forced to zero weight. Longformer, BigBird, OpenAI sparse transformer. Differential attention. Compute two attention maps with separate Q/K projections, subtract one from the other. Kills the "attention sink" that bleeds weight into the first few tokens. Microsoft's DIFF Transformer (2024). These coexist. A 2026 frontier model often mixes them: most layers are SWA-1024, every fifth is global full attention, and a handful are differential heads that clean up retrieval. Gemma 3's 5:1 SWA-to-global ratio is the…
Attention Variants — Sliding Window, Sparse, Differential: Full attention is a circle. Every token sees every token, and memory pays the price. Four variants…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.