Serving Engine Internals — PagedAttention, Continuous Batching, Chunked Prefill
Modern serving-engine throughput rests on three compounding defaults, not a single trick. PagedAttention is always on. Continuous batching injects new requests into the active batch between decode iterations. Chunked prefill slices long prompts so decode tokens never starve. Turn all three on and a Llama 3.3 70B FP8 on one H100 SXM5 pushes 2,200-2,400 tok/s at 128 concurrent — roughly 25% above vLLM's own default and 3-4x a naive PyTorch loop. This lesson reads the scheduler and attention kernel of vLLM — the reference engine for all three techniques — at a level you can diagram, and ends with a toy continuous batcher in code/main.py that schedules prefill and decode the way vLLM does. Explain PagedAttention as a KV cache allocator: blocks, block tables, and why fragmentation stays under 4% at production load. Diagram continuous batching at the iteration level: how finished sequences leave the batch and new ones join without draining. Describe chunked prefill in one sentence and name which latency metric it protects (hint: it is TTFT tail, not mean throughput). Name the 2026 vLLM v0.18.0 gotcha that bites teams enabling every optimization at once. A naive PyTorch serve loop runs one request at a time: tokenize, prefill, decode until EOS, return. At one user this works. At one hundred, it is a queue of patient people. The…
Serving Engine Internals — PagedAttention, Continuous Batching, Chunked Prefill: Modern serving-engine throughput rests on three compounding defaults, not a…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.