R.6
Index of figures
Every figure of this manual, in order, by the one claim its caption makes.
Each row links to its figure, and the claim is the claim: field of each figure block. Measured numbers in a figure come from our runs on the Mac or on the GPUs, and each section links the files it uses. Numbers from a field report name the report. The colours follow the palette of the front matter:
- blue is the SGLang server process: the tokenizer manager, the scheduler, and the detokenizer
- violet is the model and its forward pass: kernels, attention backends, and CUDA graphs
- green is the KV cache: pool slots, radix tree nodes, and cached tokens
- plum is model compute: prefill and decode steps
- olive is the client and the network
- indigo is streams and messages
- amber is a lifecycle state, such as a request state or a scheduler phase
- teal is durable state: host memory, storage tiers, and weights on disk
- rose is a failure: out of memory, retraction, an abort, or a crash
- grey is outside SGLang: the operating system, other programs, and the GPU driver
| Fig. | The claim it makes |
|---|---|
| 0.1 | Each arrow style marks one kind of exchange, and every label is a real name, message, or number from our recorded runs. |
| 1.1 | The median TTFT divided by the prompt length is about 0.32 ms per prompt token, and one output token costs 4.38 ms in decode. |
| 1.2 | SGLang owns tokenizing, scheduling, the KV cache, and the forward pass, while the model and the operator own quality, memory, and routing. |
| 1.3 | The tokenizer manager, the scheduler, and the detokenizer each touched the request once, and the prefill computed 3 of 5 tokens. |
| 2.1 | One request crosses three processes over ZMQ, and only the scheduler touches the model, so text work never runs on the thread that launches GPU steps. |
| 2.2 | Each turn ingests requests, orders the queue by policy, admits what fits the budget, and runs either one prefill batch or one decode step. |
| 2.3 | With ignore_eos, each request reserves its 256 prompt tokens and all 512 output tokens, so five requests use 3,840 tokens and a sixth would need 4,608. |
| 2.4 | The ratio starts at 0.7, falls to 0.098 after 600 decode steps, and jumps back up after a retraction, so admission grows bolder until it fails. |
| 2.5 | Without overlap the GPU waits for CPU bookkeeping after every step, and with overlap the CPU processes step N while the GPU already runs step N+1. |
| 2.6 | The scheduler decides when a request stops, the detokenizer turns ids into text, and the output streamer holds back text that could start a stop string. |
| 3.1 | One token of Qwen3-0.6B costs 114,688 bytes, the product of five numbers from the model config. |
| 3.2 | Two requests that share a system prompt point their first rows at the same KV slots, so the prefix exists once in memory. |
| 3.3 | A pool sized at 87 percent of free memory left too little for activations, and a fraction of 0.4 kept the server alive. |
| 3.4 | Raising the fraction from 0.7 to 0.82 on the GB10 grew the BF16 pool by 63 percent and changed closed-loop throughput by less than 2 percent. |
| 3.5 | The radix cache cut the shared-prefix benchmark from 129.7 to 48.8 seconds and the median first-token wait from 7,222 to 1,358 ms. |
| 3.6 | With a 4,096-token shared prompt on the L4, the radix cache cut the median TTFT from 14,706 to 479 ms. |
| 3.7 | Two running requests share one system-prompt node, and the tree splits exactly where their questions start. |
| 3.8 | A random ID at the start of the prompt made about 3,027 tokens miss the cache, and the same ID at the end cost about 45 tokens. |
| 3.9 | Under write_through, a prefix gets a host copy on its first hit, so GPU eviction no longer forces a full prefill. |
| 3.10 | At 32 distinct system prompts on the L4, HiCache cut the median TTFT from 5,449 to 817 ms. |
| 3.11 | A new request that matches 1,800 tokens of a Mamba model resumes from the last snapshot at 1,792 tokens, and prefills 8 tokens. |
| 4.1 | In eager mode the CPU launches each kernel of every layer, and in graph mode one replay call runs the whole recorded decode step. |
| 4.2 | A decode batch replays the next captured graph when it fits under max_bs, and runs eager with one launch per kernel when it does not. |
| 4.3 | SGLang picks the attention backend from the GPU generation and the attention type of the model, and each branch records a measured limit. |
| 4.4 | The checkpoint names its own quantization method, and a --quantization flag only helps when it matches that method or a known upgrade. |
| 4.5 | The FP8 KV cache is a few percent slower at every load both caches can run, and keeps scaling after the BF16 cache runs out of memory. |
| 4.6 | On the GB10, FP8 weights and FP8 KV gave more tokens per second and less time per token at every concurrency from 1 to 64. |
| 4.7 | On one L4, BF16 throughput stopped at 270 tokens per second at concurrency 32, and FP8 kept climbing to 736 at 64. |
| 4.8 | One EAGLE step drafts a chain of tokens, verifies them in one target forward, and commits the accepted prefix plus one bonus token. |
| 4.9 | Speculation doubled single-request speed and cut throughput at concurrency 20 on the same GPU and model. |
| 4.10 | On the GB10, EAGLE-3 cut the median time per token by 1.60 times at concurrency 1 and by 1.89 times at 32. |
| 4.11 | On the L4, EAGLE-3 gave 2.37 times the output throughput at concurrency 1 and 1.00 times at 32. |
| 4.12 | A schema compiles once in a thread pool while the request waits in the grammar queue, then masks the logits at every decode step. |
| 4.13 | A default kernel splits the same row differently in two batches, and a batch-invariant kernel uses one split for every batch. |
| 5.1 | Concurrency and peak concurrent requests come from request timestamps after the run, so neither line repeats the cap you set. |
| 5.2 | Raising the cap from 8 to 32 added no throughput and made the median first token five times slower. |
| 5.3 | Three seeds per point agree closely, so each line is a measurement and the gaps between the lines are real. |
| 5.4 | About half of the requests waited several times longer than the other half, so the median describes neither group. |
| 5.5 | With fixed lengths the median TTFT was 1,996 ms and P99 ITL 82 ms, and with mixed lengths they were 291 ms and 300 ms. |
| 5.6 | FP8 served 1.03 requests per second within the target at 1 offered, and BF16 served almost no request within the target at any rate. |
| 5.7 | On the L4, FP8 served 3.18 requests per second within the target at 4 offered, and BF16 served none within the target from 4 offered. |
| 5.8 | The hot sweep reported up to 28 percent less throughput than the cooled sweep, with the same command on the same card. |
| 5.9 | The gauge reported the last prefill interval at 0.942, while the counters over the whole run gave 0.636. |
| 5.10 | A result that fails the version or workload question tells you nothing about your server, however large its number is. |
| 5.11 | Every independent result in this section traces to one mechanism you can test on your own server. |
| 5.12 | At 64 users on the L4, vLLM gave the first token 1.7 times sooner and SGLang kept the P99 inter-token latency 12 times lower. |
| 5.13 | On the L4, both prefix caches kept the median TTFT under 500 ms at every shared length, and the cache-off run reached 14,706 ms. |
| 5.14 | At 4 requests per second on the L4, vLLM gave 11 percent more output and SGLang kept the P99 time per token 32 percent lower. |
| 6.1 | With --tp-size 8, --attn-dp-size 4, and --ep-size 8, the same eight GPUs form one TP group, four attention groups, and one expert group. |
| 6.2 | Round robin sends alternate requests to two worlds that share no KV cache, so a repeated prefix can miss in the other world. |
| 6.3 | One L4 reached 270 tokens per second, and the shared pair with --dp-size 2 peaked at 64 and fell to 42 as its clock fell. |
| 6.4 | Dispatch and combine move every token across GPUs twice per MoE layer, and the grouped matrix multiply in between is the only expert compute. |
| 6.5 | The time per output token grew 7.8 times while the batch grew 64 times, and each extra request can read other experts. |
| 6.6 | Wider expert parallelism won at batch 8 and lost 13 percent at batch 32, because less of its traffic stayed on NVLink. |
| 6.7 | The decode server reserves KV slots and publishes their addresses, then the prefill server writes the KV pages straight into decode memory. |
| 6.8 | Every transfer moves forward through bootstrapping, waiting, and transferring to success, and any stage can end in failure. |
| 6.9 | The gateway checks load balance first, then sends a request to the server with the longest match only when more than 30 percent of its text matches. |
| 7.1 | Only a SIGTERM to the main process drains running requests, and a forced exit or a signal to the process group stops work in progress. |
| 7.2 | Measure first, then fix memory, prompts, and concurrency, and only then touch kernels, parallelism, and disaggregation. |
| 7.3 | SGLang and the trainer never hold the GPU at the same time in this mode, and SGLang gets its KV cache back only after the new weights arrive. |
| 7.4 | A weight update is a session: pause, begin, send every bucket, end with one repack, set the version, and resume. |
| 7.5 | On one GB10, the trainer phases took 1.8 times as long as the SGLang rollout, and the weight update took under a second. |
| 7.6 | Over 48 steps the raw reward went up from a mean of 0.636 to 0.758, and the share of cut samples went down from 0.33 to 0.08. |
SGLang 0.5.22 · baa9bef · 2026-10-13 · Edition 2026.10