Count the parameters of a decoder stack by hand
RolesInference EngineeringLLM Product EngineeringAI Data Systems
The question
Count the parameters of a decoder stack with a model width of 4096, 32 layers and a SwiGLU feed-forward network at an expansion of 2.6. Then add an untied embedding and output head for a vocabulary of 128K. Where does the total land, and why is the published model of that size a little different?
Answer it first, aloud or on paper. Then open the answer.
The answer
Show short answer
Per block, attention takes 4 d^2 for the four projections and SwiGLU takes 3 d (r d), about 3 r d^2. At the given width and expansion that is about 67M plus 131M, about 198M per block, and 32 blocks give about 6.3B. The untied embedding and head add about 1B, so the stack lands near the published 8B. The real model uses grouped-query attention, which shrinks the key and value projections, and a wider FFN at an expansion of 3.5, which moves parameters from attention to the FFN while the total stays close.
Show full answer
The per-block arithmetic
attention: Q, K, V, O projections, each d x d -> 4 d^2
FFN: three matrices of d x (r d) -> 3 r d^2
norms: two vectors of length d -> negligibleWith d = 4096 and r = 2.6:
4 * 4096^2 = about 67M
3 * 2.6 * 4096^2 = about 131M
per block = about 198M
32 blocks = about 6.3BThe embeddings
An untied model has a token embedding of shape (vocab, d) and a separate head of the same size. At 128K by 4096 the pair adds about 1B. A tied model shares the matrix and saves half of that.
Why the published number differs
Two architectural choices move the count without changing the total much:
- Grouped-query attention. The key and value projections map
dton_kv_heads * d_headinstead ofdtod, so attention holds fewer than4 d^2parameters. - A wider FFN. The published model uses an expansion of about 3.5, so the FFN holds more than the
3 * 2.6 * d^2counted above.
The total still lands near 8B. The FFN holds about two thirds of the block parameters in either version.
What the count is for
Memory for the weights at a given data type, the bound on the cache budget that remains on the card, and a sanity check when a config file or a model card looks wrong.
Follow-ups
L2WorkingWhy are the norms negligible?
Show expected answer
each norm is one or two vectors of length d, thousands of parameters against millions per matrix.
L3SeniorHow does grouped-query attention change the per-block count?
Show expected answer
the K and V projections shrink to
dbyn_kv_heads * d_head, so attention holds less than4 d^2.L3SeniorWhy does an expansion of 4 with two matrices equal an expansion of about 2.6 with three?
Show expected answer
2 * 4 d^2 = 8 d^2and3 * 2.6 d^2is about8 d^2; the modern block kept the parameter budget and changed the activation.L4StaffThe team ties the embedding and the head. What changes in the count and what do you check?
Show expected answer
about half of the 1B disappears; check that the output logits still use the transpose of the embedding and that quality holds.
Wrong answers
Show the wrong answers that interviewers hear
“The parameter count grows with the sequence length.”
No weight depends on the sequence; the length changes activation memory and compute, not parameters.
“The embedding is a rounding error.”
At 128K by 4096 it is about 1B, more than a tenth of the model.
“Attention is where most of the parameters are.”
The FFN holds about two thirds of each block.
Taught in
The full transformer lesson does this count and explains the published difference.