Models

Inference and serving systems

Batching, the KV cache, prefix caching, schedulers and how to measure a serving engine.

32 of 30 planned

RolesInference Engineering

Level

32 items

  1. The two phases of one requestL1ScreenConcept6 min
  2. Why 255 tokens per second is a sloganL1ScreenConcept6 min
  3. Cached tokens at zero with a stable templateL2WorkingDebug8 min
  4. Choose tensor, pipeline or data parallelismL2WorkingDesign10 min
  5. How many users fit in the KV poolL2WorkingEstimate10 min
  6. How speculative decoding keeps the output the sameL2WorkingConcept8 min
  7. One token budget instead of two phasesL2WorkingConcept8 min
  8. Read the GPU clock before the throughputL2WorkingDebug8 min
  9. Read three scheduler log linesL2WorkingCode reading8 min
  10. Sleep, wake and weight updates for an RL loopL2WorkingConcept8 min
  11. Temperature zero gave two different answersL2WorkingConcept8 min
  12. The knee that the prefix cache hidL2WorkingDebug8 min
  13. Two clients, two numbers, one serverL2WorkingConcept8 min
  14. What an FP8 KV cache buys and costsL2WorkingTrade-off10 min
  15. What many LoRA adapters cost one serverL2WorkingDesign8 min
  16. Why one ID at the top of a prompt kills the cacheL2WorkingDebug8 min
  17. Why the KV cache is a pool of 16-token blocksL2WorkingConcept8 min
  18. Why the shared part of the prompt goes firstL2WorkingConcept8 min
  19. A larger token budget at low loadL3SeniorTrade-off10 min
  20. Acceptance by draft position, tokens per passL3SeniorEstimate8 min
  21. An admission cap decides who waitsL3SeniorTrade-off10 min
  22. Pick a concurrency cap when throughput keeps risingL3SeniorTrade-off12 min
  23. Pick the EAGLE3 draft depth for your concurrencyL3SeniorTrade-off10 min
  24. Should the prefill chunk be smallerL3SeniorTrade-off10 min
  25. Speculation won the benchmark and lost in productionL3SeniorDebug10 min
  26. Two data-parallel copies, less output than one cardL3SeniorDebug10 min
  27. What determinism costs a live serverL3SeniorTrade-off10 min
  28. When an FP8 checkpoint paysL3SeniorTrade-off10 min
  29. When prefill-decode disaggregation paysL3SeniorDesign12 min
  30. A split-server benchmark that counted empty answersL4StaffCode reading10 min
  31. Design a chat service on a fleet of L4 cardsL4StaffDesign15 min
  32. Same throughput, two choices at a full poolL4StaffTrade-off12 min

Taught in

The answers on this card come from these lessons and manuals.