Prefill-decode disaggregation: --disaggregation-mode

Your chat users see smooth streaming until someone pastes a long document. Then every stream pauses while the server prefills the document. Prefill is compute work on many tokens at once, and decode is one token per request per step. On one server, they share the same GPU steps, and the long prefill takes the step. Chunked prefill makes the pause shorter but cannot remove it.

When you finish this section, you can explain how SGLang moves a request's KV cache from a prefill server to a decode server. You can also start the smallest disaggregated setup and decide from measurements whether you need it.

Two servers, one request

Prefill-decode disaggregation (PD): two kinds of SGLang servers, started with --disaggregation-mode prefill and --disaggregation-mode decode. The default is null, which means one server does both. A prefill server computes the prompt's KV cache and the first token. A decode server receives that KV cache and generates the rest.

The SGLang docs give two reasons for the split. Prefill batches interrupt decode batches. With attention data parallelism, one group can prefill while another decodes, so the decoding group waits for the slowest group in each step. Separate servers remove both kinds of waiting. Each pool can also use its own parallel layout, for example a wide TP for prefill and attention data parallelism for decode.

How the KV cache moves

The gateway in section 6.4 starts every exchange. It picks one prefill server and one decode server. It adds three fields to the request: the prefill server's bootstrap host, its bootstrap port, and a random 63-bit number called the bootstrap room. Then it sends the same request to both servers at the same time.

Bootstrap room: the number that both servers use to find each other's half of one request. Think of it as a reservation number: the decode server books KV slots, and the prefill server writes into them.

1. The decode server creates a receiver, finds the prefill rank through the bootstrap server, and reserves KV slots for the prompt.
2. It sends the addresses of those slots to the prefill side.
3. The prefill server runs a normal prefill forward pass for the prompt.
4. It writes the KV pages into the decode server's reserved slots. With the default Mooncake backend, this is a one-sided RDMA write. Mooncake groups contiguous pages on both sides into one batch transfer.
5. The decode server sees the transfer finish and adds the request to its running batch without another prefill.
6. The client stream comes from the decode server.

The bootstrap server runs on the prefill host. Its default port is 8998. Each side tracks a transfer with five states: Failed, Bootstrapping, WaitingForInput, Transferring, and Success.

Settings that matter

All defaults come from the v0.5.22 code. mooncake_tcp exists for clusters without RDMA network cards. We did not measure its speed.

Do not send requests straight to a prefill or decode server. The scheduler asserts that a bootstrap room exists, and its error message tells you to send requests to the router instead.

Our position

Do not start with disaggregation. Start with one server, then add data parallel copies behind the gateway. Split prefill from decode only after you measured that prefill interference hurts your inter-token latency at the concurrency you serve. Disaggregation adds a second set of failures: the bootstrap step, the RDMA transfer, two timeouts, and a router that must pair servers correctly.

Even for the tail, the gain is a condition, not a default. A 2025 study compared split servers with servers that run both stages. It found that the gain is not guaranteed, and that it depends on the request load and on the path that moves the KV cache. We did not measure this ourselves. Before you keep a split, compare it with the same GPUs run as plain servers, at your real load.

A real incident shows the cost of skipping that measurement. A user ran one prefill and one decode server, each with TP4 (issue #12591). Time to first token was higher than on one TP4 server without disaggregation. The cause was an old Mooncake transfer engine. Upgrading it fixed the problem, and a maintainer also suggested SGLANG_DISAGGREGATION_QUEUE_SIZE=1 without attention data parallelism. Without the single-server baseline, nobody would have known that disaggregation made things worse.

Two more incidents, both from the router

At 64 concurrent requests, a GLM-4.5 setup logged many "Prefill bootstrap failed" and "Aborted by AbortReq" errors on the prefill side (issue #9266). Four to seven concurrent requests worked. The router's timeout for the prefill client was too short, so the router aborted requests that were still running. Pull request #9803 fixed the timeout.

A maintainer gave the general rule in a related thread (issue #9926). "Aborted by AbortReq" means that the client or the router aborted. Check timeouts before you suspect the transfer.

Novita AI found a different delay with GLM-4.7 on H200. With disaggregation and the overlap scheduler, throughput rose about 10 percent but time to first token got much worse. On prefill, the KV transfer waited until the next batch's kernel launch. Without CUDA graphs, that launch took hundreds of milliseconds for their 92-layer model. Their fix added a flag, --disaggregation-async-transfer. We could not find that flag in the v0.5.22 code, so check your version before you copy their command.

How many prefill servers for each decode server

The ratio depends on your traffic. dstack tested gpt-oss-120b on eight H200 GPUs with SGLang disaggregation and tensor parallelism only. At high concurrency, one prefill server for three decode servers won every metric, for prefill-heavy, decode-heavy, and balanced loads. At low concurrency, three to one and two to two gave a better time to first token. Their post shows charts without absolute numbers, so use it for the direction only.

At the largest scale, the SGLang team ran DeepSeek on 12 nodes of eight H100 GPUs. They reported 52.3k input and 22.3k output tokens per second per node for 2000-token inputs. They measured prefill on 4 nodes and decode on 9 nodes, in separate runs. The Mooncake team reported Kimi K2 on 128 H200 GPUs with 4 prefill and 12 decode nodes. They gave decode more nodes so that the KV pool could hold a batch of 480. Both are reports from the teams that built the system.

What happened on two L4s

We split Qwen3-8B into one prefill server on the first L4 and one decode server on the second. Our two cards have no RDMA, so we used --disaggregation-transfer-backend mooncake_tcp. The second card also ran another team's pods, so the decode pool held 2,925 tokens against 24,094 on prefill. We then ran the open-loop sweep of section 5.1 through the router.

… Decode batch, #running-req: 2, #token: 1206, token usage: 0.41, pre-allocated usage: 0.00, #prealloc-req: 19, #transfer-req: 0, #retracted-req: 0
… TCP lane queue-full rejection count: 4096
… Session 151.242.195.41:15367 failed.
Prefill transfer failed lines: 2902
Successful requests:                     240
Total generated tokens (retokenized):    0
Mean TTFT (ms):                          0.00

Decode could run only two requests at a time, and 19 more waited for decode memory. Prefill kept producing KV caches for them, and the TCP transfer queue to decode filled. Three minutes into the sweep, the transfer session failed, and it did not come back. Every later request failed its transfer.

The dangerous part came next. Each failed request still counted as a success through the router, with no text. The benchmark counted 240 successful requests out of 240 at 8 requests per second. It printed 751.85 output tokens per second, higher than the 728.61 of the FP8 server at the same rate. Only two numbers gave it away: zero retokenized output tokens and a mean TTFT of 0.00 ms.

We did not publish those throughput numbers. Our kit now refuses a run in which every request has a TTFT of 0.

We then ran the split the other way round. Decode had the full pool of 24,094 tokens, and prefill had the shared card with 2,925. From 1 to 8 users, every request came back with text. At 16 users, only 2,048 of 8,192 generated tokens came back as text. At 32 users, every request came back empty, and the kit stopped the sweep.

c8: Successful requests: 32 
c8: Total generated tokens: 4096 
c8: Total generated tokens (retokenized): 4096 
c16: Successful requests: 64 
c16: Total generated tokens: 8192 
c16: Total generated tokens (retokenized): 2048 
c32: Successful requests: 128 
c32: Total generated tokens: 16384 
c32: Total generated tokens (retokenized): 0 

The original direction failed early in both sweeps. At 2 users only 512 of 2,048 generated tokens came back as text, and at 4 users none did. The fixed-rate sweep lost most of its text from 0.5 requests per second. We have no measured result for the split under load on this pair of cards, and we do not claim one. Check Total generated tokens (retokenized) in every disaggregated benchmark before you read its throughput.

What to measure

Measure TTFT and inter-token latency on one server first, with the benchmark in section 5.1. If inter-token latency rises when long prompts arrive, disaggregation can help. After the split, watch these metrics:

• sglang:num_bootstrap_failed_reqs_total and sglang:num_transfer_failed_reqs_total, which must stay flat
• sglang:kv_transfer_latency_ms and sglang:kv_transfer_speed_gb_s, which show the cost of each move
• sglang:num_decode_prealloc_queue_reqs and sglang:num_decode_transfer_queue_reqs, which show requests that wait for memory or for a transfer.