Foundations

Transformers

Attention, positions, the transformer block, the KV cache and the cost of long context.

24 of 15 planned

RolesLLM Product EngineeringAI Evaluation and ReliabilityAI Data SystemsCustomer AI DeploymentDeveloper Experience and EducationInference Engineering

Level

24 items

  1. Count the attention work for 100 generated tokensL1ScreenEstimate6 min
  2. Explain query, key and value in one minuteL1ScreenConcept6 min
  3. Shuffle the words and attention does not noticeL1ScreenConcept6 min
  4. Why the attention scores are divided before the softmaxL1ScreenConcept6 min
  5. Absolute angles, relative distancesL2WorkingConcept8 min
  6. Attention runs, and the context has the wrong shapeL2WorkingDebug8 min
  7. Encoder, decoder or both for three productsL2WorkingTrade-off8 min
  8. One reshape is right and one scrambles the headsL2WorkingCode reading8 min
  9. Pick the attention variant for a 70B decoderL2WorkingTrade-off10 min
  10. The loss collapses and the samples are nonsenseL2WorkingDebug8 min
  11. What eight heads cost that one head does notL2WorkingConcept8 min
  12. What the skip path does and where dropout may goL2WorkingConcept8 min
  13. Where the triangle of minus infinity comes fromL2WorkingConcept8 min
  14. Why the norm moved in front of the sublayerL2WorkingTrade-off8 min
  15. A heatmap is not an explanationL3SeniorConcept10 min
  16. Count the parameters of a decoder stack by handL3SeniorEstimate10 min
  17. Is a tiled attention kernel an approximationL3SeniorConcept10 min
  18. Read the running softmax and find the fragile lineL3SeniorCode reading12 min
  19. Review a 2017-style block against today's defaultsL3SeniorDesign12 min
  20. Take a model from 8K to 128KL3SeniorDesign12 min
  21. Take-home, a character-level decoder on a laptopL3SeniorTake-home15 min
  22. A benchmark jumps between two model sizesL4StaffConcept12 min
  23. Design the attention stack for 128K context on one cardL4StaffDesign15 min
  24. Split a fixed compute budget between size and tokensL4StaffTrade-off15 min

Taught in

The answers on this card come from these lessons.