Index of figures

Every figure of this manual, in order, by the one claim it makes.

Each row links to its figure. Measured numbers in a figure come from our runs on the GPUs, and each section links the files it uses. The colours follow the palette of the front matter:

blue is the vLLM server: the API server process and the engine core
violet is the model and its forward pass: kernels, attention backends, CUDA graphs
green is the KV cache: blocks, the block pool, cached prefixes
plum is model compute: prefill and decode steps
olive is the client and the network: HTTP, the OpenAI API, the benchmark client
indigo is streams and messages between processes, and streamed tokens
amber is lifecycle: request states and scheduler steps
teal is durable state: weights on disk, host memory, offloaded KV
rose is failure: out of memory, preemption, aborts, crashes
grey is outside vLLM: the operating system, other programs, the GPU driver

Fig.	The claim it makes
1.1	Six startup lines show the version, the runner, the attention backend, the weight memory, the free KV memory, and the pool size.
1.2	A request crosses three processes: the API server tokenizes it, the engine core schedules it, and a GPU worker runs the model.
1.3	On our DGX Spark, 64 users got eight times the total output of one user, and each user waited about 7.5 times longer for every token.
2.1	Qwen3-8B holds 144 KiB of KV cache for every token, and Qwen3-30B-A3B holds 96 KiB, because it has fewer KV heads.
2.2	The same 24,096-token pool fits 2.94 users at the full context length, but 37 users who each hold 640 tokens.
2.3	The last line before the start fails tells you which of three settings to change.
2.4	With a fraction of 0.75, vLLM takes 91.3 of the 121.7 GiB that the CPU and the GPU share on a DGX Spark.
2.5	For Qwen3-8B, one GiB of KV cache holds about 7,300 tokens in BF16 and about 14,600 tokens in FP8.
2.6	With the clock locked and the same KV pool, FP8 weights gave 284.7 against 208.8 output tokens per second at 32 users.
3.1	A cap of 16 sequences matched the default up to 16 users, and at 32 users it cut output from 275 to 172 tokens per second.
3.2	With the prefix cache on, the first token stayed near 400 ms at every shared length on our L4.
3.3	A new request reuses cached blocks from the start of its prompt and stops at the first block that differs.
3.4	At 0.5 requests per second, a budget of 16,384 tokens raised the P99 ITL from 574 to 798 ms.
4.1	With the clock locked, a draft depth of 3 gave the highest output at 32 users, and depths 3 and 5 tied at one user.
4.2	On our L4, n-gram speculation cut the median time per token to about a third for one user and to 37 percent for four users.
4.3	On our L4, eager mode was 1.2 to 2.3 percent slower per token at every concurrency.
5.1	Without a cache reset, output peaked at 275 tokens per second at 32 users, which is the throughput peak and not the latency knee.
5.2	On our L4 with Qwen3-8B, vLLM and SGLang wrote the same output within 3 percent up to 32 users.
5.3	On our DGX Spark, the 99th percentile token gap of vLLM was more than twice that of SGLang at 32 users.
6.1	Three questions pick the layout, and inside one machine the link between GPUs and the split pick tensor or pipeline parallelism.
6.2	A prefill server computes the prompt, and a KV connector copies its cache to a decode server that writes the tokens.
7.1	Start from the symptom, check one number or log line, and then go to the page that fixes it.
8.1	A server with tensor parallel size 4 runs six processes: one API server, one engine core, and four GPU workers.
8.2	Each step spends 2,048 tokens: running requests get a decode token first, and a new prompt fills the rest in chunks.
8.3	The KV cache is one pool of 16-token blocks that requests take in any order, mapped by a block table for each request.
8.4	One target pass keeps the drafts up to the first rejection and adds one token, so one real request got about 1.8 tokens per pass.

vLLM 0.31.0 · db9527a · 2026-10-05 · Edition 2026.10