Rounds

Coding

Write the code that the lessons build. Each item links to a practice task with checks.

22 of 15 planned

RolesLLM Product EngineeringAgent Systems EngineeringAI Evaluation and ReliabilityAI Data SystemsDeveloper Experience and EducationInference Engineering

Level

22 items

  1. The scale factor in dot-product attentionL1ScreenCode reading6 min
  2. What temperature does to samplingL1ScreenConcept6 min
  3. Why softmax subtracts the row maximum firstL1ScreenCode reading6 min
  4. A causal mask that blocks the token itselfL2WorkingCode reading8 min
  5. A merge that lets two matches share a symbolL2WorkingCode reading8 min
  6. A training loss that reaches zero and learns nothingL2WorkingCode reading8 min
  7. Dropout that still acts during evaluationL2WorkingCode reading8 min
  8. Split heads with the right shape and the wrong numbersL2WorkingCode reading8 min
  9. Swap the learned position table for a fixed oneL2WorkingTake-home20 min
  10. Train a BPE merge list from word countsL2WorkingTake-home30 min
  11. Weight decay that went into the gradientL2WorkingCode reading10 min
  12. Write the sampling step of a generation loopL2WorkingTake-home20 min
  13. A KV cache that loses its last slotL3SeniorCode reading10 min
  14. Attention heads that never mixL3SeniorDebug10 min
  15. GQA that pairs query heads with the wrong groupL3SeniorCode reading10 min
  16. Nucleus sampling that drops the token which reaches pL3SeniorDebug10 min
  17. Swap the final LayerNorm for RMSNorm without breaking itL3SeniorCode reading10 min
  18. Sweep the peak learning rate and mark divergenceL3SeniorTake-home25 min
  19. Train the smallest language model past its baselineL3SeniorTake-home30 min
  20. Which parameters get weight decayL3SeniorCode reading10 min
  21. A byte-level tokenizer with protected special tokensL4StaffTake-home45 min
  22. Convert a multi-head checkpoint to grouped-query headsL4StaffEstimate15 min

Taught in

The answers on this card come from these lessons.