Multi-Head Attention
One attention head learns one relation at a time. Eight heads learn eight. Heads are free. Take more of them. A single self-attention head computes one attention matrix. That matrix captures one kind of relationship — usually the one that minimizes loss on whatever the training signal is. If your data has subject-verb agreement, co-reference, long-range discourse, and syntactic chunking all tangled together, a single head smears them into a single soft-max distribution and loses half the signal. The fix from the 2017 Vaswani paper: run several attention functions in parallel, each with its own Q, K, V projections, and concatenate the outputs. Each head operates in a smaller subspace of dimension dmodel / nheads. Total parameters stay the same. Expressive power goes up. Multi-head attention is the default every transformer in 2026 ships with. The only argument is about how many heads and whether keys and values share projections (Grouped-Query Attention, Multi-Query Attention, Multi-head Latent Attention). Multi-head attention splits, attends, concatenates Split. Take X of shape (N, dmodel). Project to Q, K, V each of shape (N, dmodel). Reshape to (N, nheads, dhead) where dhead = dmodel / nheads. Transpose to (nheads, N, dhead). Attend in parallel. Run scaled dot-product attention inside each head. Each head produces (N, dhead). The heads operate on different subspaces of the embedding and…
Multi-Head Attention: One attention head learns one relation at a time. Eight heads learn eight. Heads are free. Take more of them.
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.