You downloaded an open model, wrote a loop that calls generate, and put it behind an HTTP endpoint. One user gets answers fast. Ten users wait in a line, the GPU sits idle between requests, and every request recomputes the same system prompt. A serving engine exists to fix exactly that loop.

When you finish this section, you can name the two phases of every request and the four numbers a server is judged by. You can also say what SGLang controls and what it leaves to others.

Two phases with two different bottlenecks

Prefill: the forward pass over the whole prompt. It computes the keys and values of every prompt token in one batch of matrix multiplications, and it produces the first output token. Decode: the loop that produces each next token, one forward pass per token per request.

The two phases stress different parts of the GPU. Prefill has many tokens per pass, so it keeps the arithmetic units busy. Decode has one new token per request per pass. Each step reads all the weights and cached keys and values to do little arithmetic. Decode speed is set by memory bandwidth, and prefill speed is set by compute.

We measured the difference on one Mac with a 0.6B model. A 512-token prompt reached its first token in a median of 163 ms. After that, each new token took 4.38 ms on average.

Max request concurrency:                 1         
Successful requests:                     16        
…
Median TTFT (ms):                        162.98    
…
Mean TPOT (ms):                          4.38      

That is about 0.32 ms per prompt token against 4.38 ms per output token, a factor of about 14. Every design choice in this manual follows from that gap. Prefill work can be shared and skipped. Decode work can only be batched, so that one read of the weights serves many requests.

The four numbers

A server has to answer four questions, and a report that answers one of them answers nothing.

• Time to first token (TTFT): how long a user waits before the first word. Prefill and queueing set it.
• Time per output token (TPOT): the average gap between tokens after the first. Inter-token latency (ITL) is the same gap measured one token at a time, so its tail shows stalls. Decode sets both.
• Throughput: output tokens per second across all users. Batching sets it.
• Goodput: requests per second that meet a TTFT target and a TPOT target at the same time. It is the number your users feel.

The first three trade against each other. A larger batch raises throughput and makes each user's tokens slower. A large prefill chunk finishes a long prompt sooner and stalls every decode in the batch while it runs. Section 5.1 shows the trade on a real sweep, where more users bought no extra throughput and five times the wait.

SGLang's benchmark tool prints TTFT, TPOT, ITL, and throughput. It does not print goodput at v0.5.22, so you compute it from the per-request records yourself. The inference metrics lesson builds that calculation.

What SGLang owns, and what it does not

SGLang is three processes and a cache around a model. The tokenizer manager turns text into token ids. The scheduler decides which requests run in each forward pass, keeps the KV cache, and finds prompt prefixes it already computed. The detokenizer turns token ids back into text and streams it out. Section 2.1 follows a request through all three.

Three things stay outside. The model decides what the answer says. In our capture, a 0.6B model asked to name a planet answered "Ardens", which is no planet. SGLang served that answer correctly and fast, and no serving engine can make a model right.

The operator decides how much memory SGLang gets and which machine a request goes to. SGLang provides a separate router, the model gateway, for that second job, and section 6.4 covers it.

Where SGLang wins and where it loses

Every engine posts a benchmark where it wins. We read the reports by people outside the SGLang team, and they agree on a shape that follows from the mechanisms above.

SGLang wins when prompts repeat. A field report ran SGLang 0.5.6 and an 8B model on two H100 GPUs field Rawlinson 2025-12. With one user, TTFT was 583 ms against 2,141 ms for vLLM 0.11. That is the radix cache from section 3.3 skipping a prefill it already did.

SGLang ties on unique prompts and dense models. A field report ran SGLang 0.5.9 and Llama 3.3 70B in FP8 on one H100 field Spheron 2026. At 50 users it found 1,920 output tokens per second against 1,850. Both engines now run the same kernel families, so a 4% gap is noise for most teams.

SGLang loses TTFT on long prompts with its defaults. A field report on one H100 with 16K-token prompts measured a mean TTFT of 10.8 s against 1.9 s at 60 users field Jarvislabs 2026-05. The same run gave SGLang the best TPOT, 63 to 81 ms against 97 to 120 ms. SGLang splits a long prompt into chunks so that decodes keep moving, and it pays for that in first-token time.

SGLang is the reference engine for large mixture-of-experts models. SemiAnalysis made it the default engine for DeepSeek models in InferenceMAX field SemiAnalysis 2025-10. The first audited MLPerf result with SGLang ran DeepSeek-R1 on 512 MI355X GPUs field Crusoe 2026-09. Part 6 explains the parallelism that makes that work.

Our position

Choose SGLang for the shape of your traffic, never for a headline number. Agents, chat with a fixed system prompt, and retrieval with a fixed preamble all share long prefixes. For that traffic, SGLang's cache pays you back on every request. If every prompt is unique and long, measure both engines on your own prompts before you commit.

Distrust any SGLang benchmark that does not name its version. The most repeated number on the web, "SGLang 29% faster than vLLM", comes from one offline test whose page lists SGLang v0.2.3 next to vLLM 0.11 field AIMultiple 2026-01. At least five other posts copy it under other names. Section 5.3 shows how to read a benchmark before you believe it.

Sources:capture/out/bench-random-c1.txt, capture/out/generate.txt, capture/out/server-default.log; python/sglang/benchmark/serving.py and python/sglang/srt/entrypoints/http_server.py at v0.5.22; Rawlinson, "vLLM vs SGLang: Performance Benchmark on Dual H100 GPUs" (https://rawlinson.ca/articles/vllm-vs-sglang-performance-benchmark-h100, 2025-12-05); Spheron, "vLLM vs TensorRT-LLM vs SGLang" (https://www.spheron.network/blog/vllm-vs-tensorrt-llm-vs-sglang-benchmarks/); Jarvislabs, "SGLang vs vLLM: H100 Benchmarks" (https://jarvislabs.ai/blog/vllm-sglang-trtllm-comparison, 2026-05-18); SemiAnalysis, "InferenceMAX" (https://newsletter.semianalysis.com/p/inferencemax-open-source-inference, 2025-10-09); Crusoe, MLPerf Inference v6.1 (https://www.crusoe.ai/resources/blog/serving-5-75-million-tokens-per-second-crusoes-mlperf-inference-v6-1-results-on-amd-mi355x, 2026-09-16); AIMultiple, "LLM Inference Engines" (https://aimultiple.com/inference-engines, 2026-01-25).

The fastest way to understand a serving engine is to send it one small request and read everything it writes about it. Most people skip this. They start the server, see "ready", and point a load test at it. Then the first surprise in production is a log line they never read.

When you finish this section, you can start SGLang on a laptop and send one request through each API. You can explain every timing and token count in the response.

Starting the server

We ran SGLang 0.5.22 on Apple silicon through its MLX backend, with a 4-bit Qwen3-0.6B. This is the command every capture in this manual starts from.

SGLANG_USE_MLX=1 sglang serve --model-path mlx-community/Qwen3-0.6B-4bit --disable-cuda-graph --host 127.0.0.1 --port 30000 --enable-cache-report --enable-metrics --mem-fraction-static 0.4

sglang serve is the entry point that SGLang recommends at v0.5.22. The older python -m sglang.launch_server still works and prints a warning. --disable-cuda-graph is in the Apple silicon instructions and does nothing useful on a Mac, and the server says it is deprecated. --mem-fraction-static 0.4 is our choice, and section 3.2 shows the crash that made us set it.

Before the first request, the server sizes its memory and builds its cache. Read these four lines on every start.

[2026-10-16 13:58:03] Auto-sized attention KV pool: sys_available=48.49 GB, mlx_limit=51.8 GB, mlx_used=0.31 GB, kv_budget=19.39 GB, bytes_per_slot=114688, pool_size=181574
…
[2026-10-16 13:58:05] max_total_num_tokens=181574, chunked_prefill_size=4096, max_prefill_tokens=16384, max_running_requests=4096, context_len=40960, available_gpu_mem=47.96 GB
…
[2026-10-16 13:58:06] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=False hybrid_ssm=False hicache_attached=False
…
[2026-10-16 13:58:08] The server is fired up and ready to roll!

KV pool: the memory that holds the keys and values of every token the server keeps. It has room for 181,574 tokens here, at 114,688 bytes each. Section 3.1 derives that number from the model's shape. max_running_requests: the most requests the scheduler runs at once. The server computed 4096 from the pool size. Nobody set it.

The last line is printed after the server sends itself a request. SGLang posts "The capital city of France is" to its own /generate endpoint, then calls /freeze_gc. That freezes the Python garbage collector in all three processes, so the objects created at startup are never scanned again. Keep that detail in mind. It returns in the field report at the end of this section.

The request

We sent the native API a five-token prompt and asked for twelve tokens at temperature zero.

"meta_info": {
  …
  "finish_reason": {
    "type": "length",
    "length": 12
  },
  "prompt_tokens": 5,
  …
  "queue_time": 0.0002437499351799488,
  …
  "completion_tokens": 12,
  "cached_tokens": 2,
  …
  "e2e_latency": 0.9496850408613682,
  …
  "first_token_latency": 0.8885334208607674,
  …
  "decode_throughput": 179.8807619469758,

Read it field by field. The request waited 0.24 ms in the queue, because nothing else was running. Two of its five prompt tokens were already in the cache. Our prompt "The capital of France is" shares its first two tokens with the warm-up prompt "The capital city of France is". The radix cache found them and skipped them. The text "Paris, and the capital of Germany is Berlin. The capital" stopped at 12 tokens because we asked for 12, and finish_reason says length.

The scheduler logged the same request from its side.

[2026-10-16 13:58:10] Prefill batch, #new-seq: 1, #new-token: 3, #cached-token: 2, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 1.55
[2026-10-16 13:58:10] INFO:     127.0.0.1:63795 - "POST /generate HTTP/1.1" 200 OK

#new-token: 3 and #cached-token: 2 add up to the five prompt tokens. That pair is the most useful line SGLang prints. Section 3.4 uses nothing else to find a broken prompt.

The first-token time is not what you think

The first token took 889 ms. The whole request took 950 ms, so the eleven tokens after the first took about 61 ms. After the warm-up of the benchmark tool, a 512-token prompt reached its first token in a median of 163 ms on the same server. A prompt that is 100 times longer started five times faster.

The prompt length did not cause the 889 ms. The first request of a new shape pays one-time costs. On MLX, the first request that reuses a cached prefix also ran slow in section 3.4. We did not find the exact cause in the log. The lesson holds on any backend. The first request after a start measures the start, and you should throw it away.

The same question through the OpenAI API

Most clients speak the OpenAI chat API, and SGLang serves it on the same port. We asked the chat endpoint to name one planet in one word, with thinking turned off.

"message": {
  "role": "assistant",
  "content": "Ardens",
  "reasoning_content": null
},
"logprobs": null,
"finish_reason": "stop",
"matched_stop": 151645
…
"usage": {
  "prompt_tokens": 21,
  "total_tokens": 25,
  "completion_tokens": 4,
  "prompt_tokens_details": {
    "cached_tokens": 0
  },

A seven-word question became 21 prompt tokens, because the chat template wraps the message in role markers. That template cost is real. Count it when you budget context. The model said "Ardens", which is no planet. SGLang did its job: it ran the model the operator chose and returned its answer in 4 tokens.

cached_tokens appears here only because we started the server with --enable-cache-report. Without that flag, the OpenAI response reports no cached tokens at all, even when the scheduler reused thousands. Turn it on. It costs nothing, and it is the only way a client can see the cache.

Our position

Read the startup log before you send traffic, and warm the server before you measure it. The startup lines tell you the pool size, the request limit, and the cache type the server chose for you. All three were computed, and section 3.2 shows a case where the computed value crashed the server.

A field report from LinkedIn shows why the request path deserves the same attention as the kernels field LinkedIn 2026-02. Their ranking service ran a 0.6B model on 300-token inputs at 100 queries per second. The bottlenecks they found were tokenization, ZMQ messages between the processes, Python garbage collection, the GIL, and the scheduler. A scoring path that skips decode cut P99 latency from 6,220 ms to 454 ms. Garbage collection paused the server for 100 to 300 ms every few seconds. Their fix was a warm-up followed by a frozen collector, merged as pull request 9241.

That is the freeze you saw at startup.

Sources:capture/out/server-default.command.txt, capture/out/server-default.log, capture/out/generate.txt, capture/out/bench-random-c1.txt; python/sglang/srt/entrypoints/http_server.py (_execute_server_warmup, _freeze_gc_after_server_warmup) and python/sglang/srt/hardware_backend/mlx/model_runner.py (_compute_pool_size) at v0.5.22; LinkedIn Engineering, "Scaling LLM-based ranking systems with SGLang at LinkedIn" (https://www.linkedin.com/blog/engineering/ai/scaling-llm-based-ranking-systems-with-sglang-at-linkedin, 2026-02-20); pull request 9241 (https://github.com/sgl-project/sglang/pull/9241).

You start a server, send one request, and it works. Then you put real traffic on it, and the GPU sits at 40 percent while users wait. Most people blame the kernels first. LinkedIn did the work to find out, and the slow parts were the tokenizer, the messages between processes, Python garbage collection, and the GIL field LinkedIn 2026-02.

When you finish this section, you can start SGLang with sglang serve and name the three processes it starts. You can also read the startup log for the numbers that decide everything later.

The cost: one Python thread cannot do everything

A language model server does two kinds of work. GPU work is the forward pass: one prefill or decode step every few milliseconds. CPU work is everything around it: parsing HTTP, applying the chat template, tokenizing, turning token ids back into text, and checking stop strings.

The obvious design puts all of it in one Python process. That design fails under load. Python runs one thread at a time under the GIL. A 3,000-token prompt that is being tokenized holds the interpreter that must launch the next GPU step. Every millisecond of text work becomes a millisecond of idle GPU.

LinkedIn measured this on H100 GPUs with small ranking models. Batch tokenization alone moved P99 latency from 4,583 ms to 464 ms at 500 queries per second field LinkedIn 2026-02. The model did not change. Only the CPU path around it changed.

What SGLang starts

TokenizerManager: the object in the main process that receives each request, applies the chat template, tokenizes the text, and waits for the answer. The HTTP server and the TokenizerManager share one process and one asyncio loop.

Scheduler: a separate process that owns the GPU. It decides which tokens run next, builds the batch, runs the forward pass, and samples. With tensor parallelism there is one scheduler process per GPU.

DetokenizerManager: a third process that turns output token ids into text, one small piece at a time, and sends the text back to the TokenizerManager.

The processes talk over ZMQ sockets. The TokenizerManager sends a TokenizedGenerateReqInput to the scheduler. The scheduler sends a BatchTokenIDOutput to the detokenizer. The detokenizer sends a BatchStrOutput back. At v0.5.22 the messages are pickled by default, because SGLANG_USE_PICKLE_IPC is true code.

With tensor parallelism, only rank 0 of the scheduler opens the ZMQ sockets. It broadcasts each request to the other ranks, so every GPU builds the same batch code. No rank decides on its own.

The command we ran

We started the server on an Apple silicon Mac with the MLX backend and a 4-bit Qwen3 0.6B model. sglang serve is the recommended command at v0.5.22. The older python -m sglang.launch_server still works and prints a warning that names sglang serve as the recommended entrypoint code.

SGLANG_USE_MLX=1 sglang serve --model-path mlx-community/Qwen3-0.6B-4bit --disable-cuda-graph --host 127.0.0.1 --port 30000 --enable-cache-report --enable-metrics --mem-fraction-static 0.4

Two flags are not defaults, and both are deliberate. --enable-cache-report puts the cached token count into every OpenAI-style response, so section 2.5 can show it. --mem-fraction-static 0.4 limits the KV pool, and section 3.2 explains the crash we had without it.

The log objects to one flag. The Apple silicon page of the SGLang docs tells you to pass --disable-cuda-graph. The second line of our log says the flag "is deprecated and will be removed in a future release", and names --cuda-graph-backend-{decode,prefill}=disabled as the replacement.

The docs and the code disagree here, and the code wins. The flag still works at v0.5.22. Plan to replace it before the next release removes it.

Read the startup log before anything else

The startup log tells you what SGLang decided on your behalf. Most of those numbers are not flags you set. They are values that SGLang computed from your memory, your model, and your hardware.

[2026-10-16 13:58:05] max_total_num_tokens=181574, chunked_prefill_size=4096, max_prefill_tokens=16384, max_running_requests=4096, context_len=40960, available_gpu_mem=47.96 GB
…
[2026-10-16 13:58:06] Using the Python TreeCore: the Rust TreeCore supports Linux only
[2026-10-16 13:58:06] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=False hybrid_ssm=False hicache_attached=False
[2026-10-16 13:58:06] Engine startup timings (s): load_weight=0.00, kv_cache_allocation=0.00, scheduler_e2e=7.01, cuda_graph={prefill=0.00, decode=0.00, target_verify=0.00, draft_prefill=0.00, draft_decode=0.00, draft_extend=0.00}, tokenizer_e2e=11.92

Read the first line field by field:

• max_total_num_tokens=181574 is the size of the KV pool in tokens. Part 3 shows where it comes from.
• chunked_prefill_size=4096 is the largest piece of one prompt that runs in one step. A longer prompt runs in pieces.
• max_running_requests=4096 is how many requests can decode at the same time. It is a computed value, not a promise of good latency.
• context_len=40960 is the longest sequence the model accepts.

The tree cache lines matter too. On Linux the radix tree runs in Rust by default since v0.5.21. On our Mac it runs in Python, and the log says so.

The warm-up and the garbage collector

Before the server says it is ready, it sends itself one request. Then it calls its own /freeze_gc endpoint.

[2026-10-16 13:58:08] Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 3.85
[2026-10-16 13:58:08] INFO:     127.0.0.1:63781 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:58:08] INFO:     127.0.0.1:63785 - "POST /freeze_gc HTTP/1.1" 200 OK
[2026-10-16 13:58:08] The server is fired up and ready to roll!

That /freeze_gc call has a history. LinkedIn saw garbage collection pauses of 100 to 300 ms every few seconds and fixed them with a warm-up and gc.freeze() field LinkedIn 2026-02. At v0.5.22 the server does the same thing for you after its warm-up, for the scheduler and the detokenizer too code.

The processes in the shutdown log

The clearest proof of the process split comes when the server stops. Our capture kit stops each server with SIGTERM.

[2026-10-16 14:01:45] SIGTERM received. signum=None frame=None. Draining requests and shutting down...
[2026-10-16 14:01:45] Subprocess scheduler_0 (pid=5909) crashed with exit code -15. Triggering SIGQUIT for cleanup...

The main process is pid 5782 and the scheduler is pid 5909. Exit code -15 is our own SIGTERM, so the word "crashed" is misleading. Learn to read this line, because a real scheduler crash prints the same shape with a different exit code, as section 2.4 shows.

Our position

Treat the CPU path as part of the model server. When time to first token is high and the GPU is not busy, look at the tokenizer process before you look at kernels. SGLang gives you more workers for both ends: --tokenizer-worker-num and --detokenizer-worker-num, both 1 by default code.

Do not run every experiment on the defaults that the startup log reports. Write the computed limits down, then decide which ones to set yourself. Section 7.1 gives the order.

Sources:python/sglang/srt/entrypoints/engine.py (Engine docstring and process launch), python/sglang/srt/managers/io_struct.py, python/sglang/srt/environ.py (SGLANG_USE_PICKLE_IPC), python/sglang/srt/managers/scheduler_components/request_receiver.py, python/sglang/launch_server.py, python/sglang/srt/entrypoints/http_server.py (warm-up and /freeze_gc), python/sglang/srt/arg_groups/fields/serving.py at v0.5.22; docs/docs/hardware-platforms/apple_metal.mdx at v0.5.22; capture/out/server-default.command.txt and capture/out/server-default.log; LinkedIn Engineering, "Scaling LLM-based ranking systems with SGLang at LinkedIn", 2026-02-20, https://www.linkedin.com/blog/engineering/ai/scaling-llm-based-ranking-systems-with-sglang-at-linkedin.

Eight users send prompts in the same second. One of them pasted a 4,000-token document, and the others typed a sentence each. Should the long prompt go first, or the short ones? Should the request that shares a system prompt with yesterday's traffic jump the queue? Somebody has to decide, on every step, and in SGLang that somebody is the scheduler loop.

When you finish this section, you can read the scheduler's Prefill batch and Decode batch log lines. You can name the policy that orders the waiting queue and decide whether to change it.

The cost of a bad order

A GPU step costs about the same whether it computes tokens for 1 request or for 8. So the scheduler wants big batches. A user who waits in the queue sees nothing at all. So the scheduler also wants short queues. Every policy is a trade between those two costs.

The obvious fix is a smart order, for example the request with the most cached prefix first. The SGLang paper proposed exactly that, longest prefix match, and proved it optimal for an offline batch paper. Online traffic is not an offline batch, and the smart order costs CPU time on every step.

One turn of the loop

Scheduler loop: the function that runs forever in the scheduler process. Each turn takes new requests, picks the next batch, runs it on the GPU, and processes the results.

One turn does four things in order code:

1. Ingest. Drain new requests from the ZMQ socket and add them to the waiting queue.
2. Pick. If the last batch was a prefill, merge it into the running decode batch. Then try to build a new prefill batch from the waiting queue. If one exists, run it. If not, run one decode step for every running request.
3. Run. Build a ForwardBatch from the ScheduleBatch and run the model.
4. Process. Append the sampled tokens, check stop conditions, free or cache the KV of finished requests, and send results to the detokenizer.

A request rides one prefill batch, possibly over several chunks, and then joins the long-lived decode batch until it stops. Prefill wins when the scheduler can admit something. That rule keeps time to first token low, and section 2.3 shows the budget check that decides "can admit".

The two log lines

The scheduler prints one line per prefill batch and one decode line every 40 decode steps, the default of --decode-log-interval code. These lines are the most useful text the server writes. Here are three consecutive lines from our benchmark at concurrency 8.

[2026-10-16 13:59:07] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 1, token usage: 0.00, #running-req: 0, #queue-req: 7, #pending-token: 0, cuda graph: False, input throughput (token/s): 155.03
[2026-10-16 13:59:09] Prefill batch, #new-seq: 7, #new-token: 3553, #cached-token: 2, token usage: 0.02, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 1914.13
[2026-10-16 13:59:09] Decode batch, #running-req: 8, #token: 4075, token usage: 0.02, cuda graph: False, gen throughput (token/s): 98.37, #queue-req: 0

Read the fields:

• #new-seq is how many requests this prefill admits. #new-token is how many prompt tokens it computes, and #cached-token is how many it reused from the radix cache.
• token usage is the share of the KV pool in use. Our pool is large, so usage stays near zero here.
• #running-req counts requests that are decoding. #queue-req counts requests that are waiting.
• cuda graph: False is expected on a Mac. On CUDA, True means the decode step replayed a captured graph, and section 4.1 explains why that matters.

The first line admits one request while seven wait. The next prefill admits the other seven together, 3,553 new tokens in one step at 1,914 tokens per second. Then all eight decode as one batch. Batching prefill is where the throughput comes from, and the log shows it happen.

The policies

Schedule policy: the rule that orders the waiting queue before admission. Set it with --schedule-policy.

The default at v0.5.22 is fcfs code. The choices split into two groups.

Many readers expect lpm, because the paper describes it. It was the default in v0.4.3 and became fcfs by v0.4.6. The research for this manual did not find the pull request that changed it.

Two rules in the code limit the cache-aware policies code:

• When the waiting queue holds more than 128 requests, lpm and hrrn change to FCFS for that turn. The comment says why: "Turn off the expensive prefix matching and sorting when the #queue is large."
• When the radix cache is off, every cache-aware policy becomes FCFS.

So lpm helps only in a narrow band. The queue must be long enough to reorder, short enough to stay under 128, and full of shared prefixes.

Our position

Keep fcfs. It is fair, it costs no prefix matching on the hot path, and the radix cache still gives you the cache hits. A shared prefix that arrived first is in the tree when the next request matches it, whatever the order.

Switch to lpm only when three things are true. Your traffic shares long prefixes. Your queue is often between a few and 128 requests deep. You measured the cache hit rate before and after with /metrics. Section 5.2 shows the cache_hit_rate series to compare.

If you need tiers of users, use --enable-priority-scheduling instead of a cache policy. A new request can then preempt running ones when its priority beats theirs by --priority-scheduling-preemption-threshold, 10 by default code.

An incident: the limit you set is not the limit you get

A user served a hybrid Mamba model and set --max-running-requests 8 issue #36889. The server accepted the flag and reported 8. Throughput flattened at 2 streams anyway. Only an info line in the log said that the Mamba state cache capped max_running_requests to 2.

The scheduler loop admits against every limit at once, and the smallest one wins. The fix the reporter used was more Mamba cache, --max-mamba-cache-size 40 with bf16 state and --mem-fraction-static 0.90, for 75 percent more throughput at concurrency 12. At v0.5.22 the cap is a warning. Read the log for the limit that the loop really uses, not the one you typed.

Sources:python/sglang/srt/managers/scheduler.py (run_event_loop, event_loop_normal, get_next_batch_to_run, run_batch, process_batch_result), python/sglang/srt/managers/schedule_policy.py (CacheAwarePolicy, CacheAgnosticPolicy, WAITING_QUEUE_PREFIX_MATCH_MAX, _determine_active_policy, _validate_and_adjust_policy), python/sglang/srt/arg_groups/fields/schedule.py at v0.5.22; SGLang paper, arXiv 2312.07104; capture/out/server-default.log; issue https://github.com/sgl-project/sglang/issues/36889.

Your dashboard says the GPU has free memory, yet new requests wait 28 seconds for their first token. Nothing crashed, nothing was dropped, and the per-token speed is fine. Users still feel that the service is down. This is what admission control looks like when it does its job, and you need to know why it chose to make them wait.

When you finish this section, you can explain the token budget that decides admission and compute how many requests fit in a pool. You can also tell a healthy queue from a retraction problem in the log.

The cost: memory you will need later

A request needs KV cache for its prompt now, and for every token it generates later. The scheduler does not know how many tokens a request will generate. It knows only max_new_tokens, the upper limit.

There are two obvious fixes, and both fail. Reserve the full max_new_tokens for everyone, and most of the pool sits empty, because most requests stop early at an end-of-sequence token. Reserve nothing, and the pool fills during decode, and the server must throw away work.

The forecast

SGLang uses a forecast with a correction. Two rules decide admission code:

• A new request must fit its uncached prompt tokens plus its remaining max_new_tokens, clipped to 4,096 by SGLANG_CLIP_MAX_NEW_TOKENS_ESTIMATION.
• Each running request holds back its remaining max_new_tokens times the new token ratio.

New token ratio: the share of remaining output that the scheduler expects a running request to use. It starts at 0.7 times --schedule-conservativeness (1.0 by default). It falls linearly to 0.14 of its start value, which is 0.098 at defaults, over 600 decode steps code.

The ratio falls because most requests stop early. The scheduler grows bolder the longer things go well. A request with ignore_eos set cannot stop early, so its ratio is always 1.0.

What we measured with a small pool

We started a server with --max-total-tokens 4096, a pool far smaller than our Mac allows. Then we ran the benchmark at concurrency 16 with 256-token prompts and 512-token outputs.

The benchmark client sets ignore_eos to true by default, so every request generates exactly 512 tokens code. Each request therefore reserves 256 plus 512, which is 768 tokens. Five requests reserve 3,840 tokens, and a sixth would need 4,608. The scheduler admitted exactly five.

[2026-10-16 14:08:02] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 255, token usage: 0.06, #running-req: 0, #queue-req: 15, #pending-token: 0, cuda graph: False, input throughput (token/s): 0.80
[2026-10-16 14:08:02] Prefill batch, #new-seq: 4, #new-token: 896, #cached-token: 0, token usage: 0.28, #running-req: 1, #queue-req: 11, #pending-token: 2808, cuda graph: False, input throughput (token/s): 3889.60
…
[2026-10-16 14:08:14] Decode batch, #running-req: 5, #token: 1517, token usage: 0.37, cuda graph: False, gen throughput (token/s): 170.33, #queue-req: 11
[2026-10-16 14:08:17] Decode batch, #running-req: 5, #token: 2117, token usage: 0.52, cuda graph: False, gen throughput (token/s): 205.04, #queue-req: 11
…
[2026-10-16 14:08:26] Decode batch, #running-req: 5, #token: 3717, token usage: 0.91, cuda graph: False, gen throughput (token/s): 139.02, #queue-req: 11

Watch token usage. It climbs from 0.28 to 0.91 as the five requests generate into the space that the forecast reserved for them. Eleven requests wait the whole time. When the five finish, the next five start, and the log repeats in waves.

The client saw the cost of this order:

Successful requests:                     32        
…
Concurrency:                             13.06     
…
Median TTFT (ms):                        27943.11  
…
Mean TPOT (ms):                          24.26     

Every request succeeded. Once a request started, each token took 24 ms. The median user still waited 27.9 seconds for the first token. The Concurrency value of 13.06 is the client's average number of open requests. Only five of them were on the GPU at any time.

Retraction: when the forecast is wrong

The forecast can be too bold, because the ratio decays to 0.098. Requests that run long then need more slots than the pool has. Before each decode step, the scheduler checks that every running request can get one more slot code.

Retraction: removing running requests from the decode batch to free their KV slots, and putting them back in the waiting queue. They prefill again later.

When the check fails, SGLang logs KV cache pool is full. Retract requests.. It retracts the requests with the fewest output tokens first, and among those the longest prompts first, because they lost the least work code. It resets the ratio to a new estimate: tokens decoded so far plus 20 per request, divided by the total max_new_tokens. At v0.5.22, if even the last request cannot fit, it aborts that request with "Out of memory even after retracting all other requests" instead of looping.

Our small-pool run had no retractions at all. The metrics endpoint confirms it.

sglang:num_retracted_reqs{engine_type="unified",model_name="mlx-community/Qwen3-0.6B-4bit",moe_ep_rank="0",pid="38678",pp_rank="0",tp_rank="0"} 0.0

The forecast had nothing to correct. With ignore_eos, the reservation was exact. Real traffic stops early, so the ratio decays and admission becomes bolder, and retraction becomes possible.

Incidents: when retraction is the problem

A server running DeepSeek-R1 on eight H20 GPUs logged KV cache pool is full. Retract requests. on every rank, again and again, and then stopped answering health checks issue #8463. A commenter traced it to retraction that could not free enough memory and repeated forever. The issue closed without a fix. The abort of the last request at v0.5.22 addresses that loop.

Another user saw decode throughput fall from about 429 to between 20 and 38 tokens per second after a retraction, and stay there issue #26248. That issue closed for inactivity with no recorded cause. The lesson stands: a retraction in the log is a performance alarm.

Our position

Queueing is the honest failure. A queue keeps every running request fast and makes the overload visible in #queue-req and in time to first token. Retraction hides overload as wasted prefill.

The SGLang tuning docs say an occasional retraction, about one per minute, is fine. Treat more than that as a configuration bug. Raise --schedule-conservativeness, for example to 1.3, so the forecast starts higher. Or lower --max-running-requests. Do not fix it with a larger --mem-fraction-static alone, because section 3.2 shows what that costs.

Set realistic max_new_tokens in your clients. A client that asks for 4,096 tokens and uses 200 makes every new request look about 20 times bigger than it is.

Sources:python/sglang/srt/managers/schedule_policy.py (PrefillAdder.add_one_req, _get_running_request_total_token_offset, CLIP_MAX_NEW_TOKENS), python/sglang/srt/managers/scheduler_components/new_token_ratio_tracker.py, python/sglang/srt/managers/schedule_batch.py (check_decode_mem, retract_decode, _get_decode_retraction_order), python/sglang/srt/managers/scheduler.py (update_running_batch), python/sglang/srt/environ.py, python/sglang/benchmark/serving.py (ignore_eos default) at v0.5.22; docs/docs/advanced_features/hyperparameter_tuning.mdx at v0.5.22; capture/out/server-small-pool.command.txt, server-small-pool.log, small-pool-bench-long-output-c16.txt, small-pool-metrics.txt; issues https://github.com/sgl-project/sglang/issues/8463 and https://github.com/sgl-project/sglang/issues/26248.

Profile a simple serving loop and you find gaps on the GPU timeline between decode steps. In each gap the GPU waits while Python reads the sampled tokens, checks stop strings, frees memory, and builds the next batch. A decode step on a small model takes a few milliseconds, so a gap of one millisecond is a large share of the step.

When you finish this section, you can explain how SGLang hides the CPU work between steps and say when the overlap is off. You can decide whether --disable-overlap-schedule belongs in your command.

The cost: CPU time between GPU steps

Overlap scheduling: running the CPU work for one batch at the same time as the GPU work of another batch.

The SGLang team described the problem in the v0.4 release post. Batch scheduling, memory allocation, and prefix matching leave the GPU idle between steps team 2024-12. Their answer: "The scheduler runs one batch ahead and prepares all the metadata required for the next batch." They report a 1.1x throughput gain from it team 2024-12. That number is their claim, measured on their GPUs.

The obvious fix is a second thread that builds batches while the first one runs the GPU. Python threads share one interpreter, so a scheduler thread and a forward thread take turns. SGLang tried a thread first. Since pull request #11210 in October 2025 it uses two CUDA streams in one thread instead code.

How it works on CUDA

The loop launches batch N+1 on the GPU first, and then processes the results of batch N on the CPU code. The GPU never waits for the bookkeeping of the step before.

There is a problem. Batch N+1 needs the tokens that batch N sampled, and those tokens are still on the GPU when N+1 is built. SGLang builds N+1 with placeholder token ids. A FutureMap fills in the placeholders on the GPU from the output of N. The CPU never needs to read the tokens before the launch code.

How it works on a Mac

The MLX backend has its own overlap loop, event_loop_overlap_mlx code. MLX evaluates lazily, so the loop keeps two decode graphs queued with mx.async_eval. The docstring says the GPU executes the next step "back-to-back" with the current one. It adds: "there is no scheduling gap on the device." The chain breaks when a prefill arrives, a request finishes, or the waiting queue has new work. Then the loop builds a fresh step.

The scheduler is the same Python scheduler on CUDA and on MLX. The batching, the radix tree bookkeeping, and the log lines in section 2.2 are the same. Only the forward pass and the overlap mechanism differ.

What happened when we turned it off

The Apple silicon page of the SGLang docs lists --disable-overlap-schedule as one of the key parameters for the MLX backend. We added it to our server command and ran the concurrency 8 benchmark. The server answered its own warm-up and the benchmark warm-up. Then the scheduler died on the first batch of the real run.

[2026-10-16 14:07:15] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 511, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 0.39
[2026-10-16 14:07:16] Scheduler hit an exception: Traceback (most recent call last):
…
    scheduler.event_loop_normal()
…
  File "/private/tmp/sglang-research/sglang-v0.5.22/python/sglang/srt/managers/scheduler.py", line 4746, in run_batch
    self._relay_forward_payload(
  File "/private/tmp/sglang-research/sglang-v0.5.22/python/sglang/srt/managers/scheduler.py", line 4852, in _relay_forward_payload
    self.future_map.stash(future_indices, payload)
  File "/private/tmp/sglang-research/sglang-v0.5.22/python/sglang/srt/managers/overlap_utils.py", line 639, in stash
    self.output_tokens_buf[indices] = payload.bonus_tokens.to(
…
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, mps:0 and cpu!

Read the stack from the bottom. With the overlap off, the dispatcher picks event_loop_normal, the generic loop. That loop calls run_batch, which still relays the sampled tokens through the FutureMap of the overlap machinery. On MLX the buffer lives on the Metal device and the sampled tokens arrive on the CPU, and PyTorch refuses to mix them. The overlap loop for MLX does not take this path, so the default server never reaches it.

The main process then shut everything down. Every one of the 32 benchmark requests failed: 8 were cut off during their response, and 24 could not connect after the server was gone.

Warmup completed with 1 sequences. Starting main benchmark run...
…
requests.exceptions.ConnectionError: HTTPConnectionPool(host='0.0.0.0', port=30000): Max retries exceeded with url: /server_info (Caused by NewConnectionError("HTTPConnection(host='0.0.0.0', port=30000): Failed to establish a new connection: [Errno 61] Connection refused"))

Our capture kit runs this configuration on every recording, and it fails the same way every time. We report it as we found it: a flag that the platform docs recommend crashes the platform at the pinned release.

A GPU incident with the same root

On CUDA, a user ran with overlap on and hit an allocation failure right after KV cache pool is full at 94 percent token usage issue #14972. Under overlap, the next batch is planned while the tokens of the previous one are still being computed. Retraction had to reason about requests that would finish one step later. A fix proposed in pull request #15626 closed without merge. At v0.5.22 the scheduler removes requests that will finish by length from the next decode batch. It also skips retraction in a step where finishing requests will free KV code.

Both stories have one lesson. Overlap is a second copy of the timeline in the scheduler's head, and every feature has to agree with it.

Our position

Leave the overlap on. It is the default, and it hides CPU time that you would otherwise pay on every step. At v0.5.22 it is also the path that the developers and most users run.

Turn it off for one reason: to rule it out while you debug a wrong output or a hang. Expect the off path to be less tested, and test it on your hardware before you depend on it. SGLang itself turns the overlap off where it cannot work, for example on Apple MPS without MLX and for speculative decoding on CPU code.

Sources:python/sglang/srt/managers/scheduler.py (event_loop_overlap, event_loop_normal, _dispatch_event_loop_once, run_batch, _relay_forward_payload, update_running_batch), python/sglang/srt/managers/overlap_utils.py (FutureMap.stash), python/sglang/srt/hardware_backend/mlx/scheduler_mixin.py (event_loop_overlap_mlx) at v0.5.22; docs/docs/hardware-platforms/apple_metal.mdx at v0.5.22; LMSYS, "SGLang v0.4: Zero-Overhead Batch Scheduler", 2024-12-04, https://lmsys.org/blog/2024-12-04-sglang-v0-4/; pull request https://github.com/sgl-project/sglang/pull/11210; issue https://github.com/sgl-project/sglang/issues/14972 and pull request https://github.com/sgl-project/sglang/pull/15626; capture/out/server-no-overlap.command.txt, server-no-overlap.log, no-overlap-bench-random-c8.txt.

You stream a reply to a chat window, and the last word before your stop string arrives late. Or the stop string itself leaks into the text when speculative decoding is on. Or the client shows no cached_tokens at all, so you cannot tell whether your prompt hit the cache. All three are decisions about where text is made and who reports what.

When you finish this section, you can choose between the native /generate endpoint and the OpenAI endpoints. You can read every timing field in a native response and explain why streamed text sometimes waits.

Two APIs on one server

Native API: SGLang's own endpoints, mainly POST /generate. It takes text or token ids and sampling_params, and it returns a meta_info object with SGLang's own data.

OpenAI API: the /v1/chat/completions, /v1/completions, and /v1/models endpoints, in the format that OpenAI clients expect. The server applies the chat template before it tokenizes.

Both endpoints build the same internal request and cross the same three processes from section 2.1. The difference is what comes back.

What the native response tells you

Our capture kit sends one native request with 12 output tokens at temperature 0.

{
  "text": " Paris, and the capital of Germany is Berlin. The capital",
  …
  "meta_info": {
    "id": "e272e3ffc4064bfeaa0223b86408e107",
    "finish_reason": {
      "type": "length",
      "length": 12
    },
    "prompt_tokens": 5,
    …
    "num_retractions": 0,
    …
    "queue_time": 0.0002437499351799488,
    "reasoning_tokens": 0,
    "completion_tokens": 12,
    "cached_tokens": 2,
    …
    "cached_tokens_details": {
      "device": 2,
      "host": 0
    },
    "dp_rank": null,
    "e2e_latency": 0.9496850408613682,
    …
    "first_token_latency": 0.8885334208607674,
    …
    "decode_throughput": 179.8807619469758,
    …
  }
}

Every field answers a question you will ask in production:

• finish_reason says why it stopped: length here, because the request asked for 12 tokens and got 12.
• queue_time is 0.24 ms, so the request did not wait. Under load this number is the first one to grow.
• first_token_latency is 889 ms and e2e_latency is 950 ms. The first token cost most of the time on a cold server.
• num_retractions is 0. Section 2.3 explains what a nonzero value costs.
• cached_tokens is 2 of 5 prompt tokens, all on the device and none from host memory.

Two cached tokens on the first request of our kit is a small mystery with a clean answer. Before it accepts traffic, the server warms up with the prompt "The capital city of France is" code. Our prompt was "The capital of France is". The first two tokens matched the warm-up in the radix tree. The cache does not care which user sent a prefix.

What the OpenAI response tells you

  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Ardens",
        "reasoning_content": null
      },
      "logprobs": null,
      "finish_reason": "stop",
      "matched_stop": 151645
    }
  ],
  "usage": {
    "prompt_tokens": 21,
    "total_tokens": 25,
    "completion_tokens": 4,
    "prompt_tokens_details": {
      "cached_tokens": 0
    },
    "reasoning_tokens": 0
  },

We asked for one planet in one word. The 4-bit 0.6B model answered "Ardens", which is not a planet. Keep that in mind when you judge a serving engine with a tiny model. The engine did its job: it stopped at token 151645, the end-of-turn token, and reported matched_stop.

The cached_tokens field is there only because our server runs with --enable-cache-report. Without the flag, SGLang omits the cache details from OpenAI usage, even though it computes them for every request code. The native meta_info reports cached_tokens either way.

A note for Mac readers. On MLX at v0.5.22, decode is greedy unless you pass --mlx-enable-sampling, so temperature, grammar masks, and output logprobs have no effect there. The help text of sglang serve says so. Our kit uses temperature 0, so its outputs are the same either way. Section 4.6 covers sampling and logprobs on GPUs.

Where stop strings are checked

Stop condition: a rule that ends a request. It can be a stop token, a stop string, a regex, the max_new_tokens limit, an abort, or the end of a grammar.

SGLang checks stop conditions in the scheduler, after each step, in a fixed order code. A stop string is checked before the end-of-sequence token. The code comment gives the reason. Speculative decoding can accept more than one token in a step. A token-based check "would trim only the last token and leak the stop string." The detokenizer only trims the matched stop text from the output.

Why streamed text waits

Two rules can delay a chunk code:

• Incomplete characters. The detokenizer decodes incrementally with two offsets per request. If the new text ends in an incomplete UTF-8 character, it waits for the next token before it sends anything.
• A possible stop string. If the tail of the text could be the start of a stop string, the output streamer holds it back. If the next tokens complete the stop string, the text never appears. If they do not, the held text goes out with the next chunk.

With the default --stream-interval 1, SGLang streams after every step. A non-streaming request still sends progress inside the server every 50 tokens, SGLANG_FORCE_STREAM_INTERVAL, so the TokenizerManager can track it code.

An incident: the detokenizer that never moved

The incomplete-character rule has a cost when the tokens never complete a character issue #34740. The detokenizer waits while the text ends in the replacement character U+FFFD. Byte tokens that never complete a character never advance the offsets. Every step then decodes the whole tail again, and the work grows with the square of the length. The report found it with a test setting that fills id 100, which decodes to U+FFFD on DeepSeek-V4-Pro. The issue was open when we checked.

Our position

Use the native /generate endpoint when you debug, measure, or build your own client. Its meta_info tells you queue time, first token latency, cache hits, and retractions per request. The OpenAI format has no place for most of that.

Use the OpenAI endpoints for applications that already speak that format. Start every server with --enable-cache-report, so those clients can see cache hits too. It costs nothing, and part 3 shows how much the number is worth.

Sources:python/sglang/srt/entrypoints/http_server.py (warm-up prompt, /generate), python/sglang/srt/entrypoints/openai/serving_chat.py (enable_cache_report), python/sglang/srt/managers/schedule_batch.py (update_finish_state, check_match_stop_str_prefix), python/sglang/srt/managers/scheduler_components/output_streamer.py, python/sglang/srt/managers/detokenizer_manager.py, python/sglang/srt/environ.py (SGLANG_FORCE_STREAM_INTERVAL), python/sglang/srt/arg_groups/fields/serving.py at v0.5.22; capture/out/generate.txt and capture/one_request.py; capture/out/serve-help.txt for --mlx-enable-sampling; issue https://github.com/sgl-project/sglang/issues/34740.

You rent a GPU with 80 GB of memory for an 8-billion-parameter model. The weights take about 16 GB, so most people expect room for a large number of users. Then the server queues requests at a few dozen concurrent conversations, and the GPU still shows free compute. The weights did not fill the memory. The keys and values of every token did.

When you finish this section, you can compute the KV cost of a model from its config. You can find the same number in the startup log and explain why SGLang counts capacity in tokens.

Why the cache exists

KV cache: the stored keys and values of every earlier token, for every layer, kept so that decode does not compute them again. A decode step makes one new token. That token attends to the keys and values of all the tokens before it. Without the cache, each new token would repeat the whole prefill of the conversation.

So the cache grows with two things: the length of each conversation and the number of conversations that run at the same time. After the weights, it is the largest user of memory in a serving engine. Every scheduling decision in Part 2 is a decision about this memory.

The formula, and our first measured number

For a model with ordinary multi-head or grouped-query attention, one token costs this many bytes:

2 × layers × KV heads × head dim × bytes per value

The 2 counts the key and the value. Grouped-query attention: several query heads share one key and value head, which is why modern models have far fewer KV heads than attention heads. SGLang computes this number when it starts code. On our Mac, the MLX backend printed it in the first lines of the log.

[2026-10-16 13:58:03] Auto-sized attention KV pool: sys_available=48.49 GB, mlx_limit=51.8 GB, mlx_used=0.31 GB, kv_budget=19.39 GB, bytes_per_slot=114688, pool_size=181574
…
[2026-10-16 13:58:08] MlxAttentionKVPool: 181575 slots x 28 layers x 8 heads x 128 dim, dtype=mlx.core.bfloat16, ~19859.8 MB

We measured bytes_per_slot=114688 for Qwen3-0.6B. The second line gives every term: 28 layers, 8 KV heads, a head dimension of 128, and bfloat16, which is 2 bytes. So the cost is 2 × 28 × 8 × 128 × 2, which is 114,688 bytes, or 112 KiB per token. The formula and the log agree to the byte.

The same arithmetic on public model configs gives these costs:

Two results surprise most people. A 0.6B model costs almost as much per token as an 8B model. The layer count and the KV heads decide the cost, and the parameter count does not. A 100,000-token conversation on Llama-3.1-8B holds 12.5 GiB of KV, which is close to the size of the weights.

Multi-head latent attention (MLA): DeepSeek's attention, which stores one compressed latent per token instead of separate keys and values per head. MLA is small per token, but every tensor-parallel rank stores the full latent code. Add GPUs with tensor parallelism and you do not add tokens. Section 6.1 shows how data-parallel attention answers that problem.

Two tables, one pool

SGLang keeps the KV cache in a pool of fixed-size slots, one slot per token. Two tables sit in front of it, and they answer different questions code.

ReqToTokenPool: a table with one row per running request and one column per token position. Each cell holds the index of a slot in the KV pool. Row 0 is padding, so that padded batch entries write to a harmless place.

TokenToKVPoolAllocator: the free list of KV slots. It gives out slot indices when a request needs room and takes them back when the request ends. Slot 0 is reserved for the same padding reason. That is why our log shows a buffer of 181,575 slots for a pool of 181,574.

The physical pool holds the tensors. Attention kernels read the keys and values through the request's row in ReqToTokenPool. This design has one large effect. Two requests that share a prefix can point their rows at the same slots, so the shared tokens exist once in memory. Section 3.3 shows the radix tree that finds those shared slots.

The cache is the pool

SGLang has no separate setting for cache size. The SGLang paper states the design directly: "we let the cached tokens and the currently running requests share the same memory pool" paper. A cached prefix is KV that no running request locks right now. When a new request needs room, SGLang evicts prefixes that no request locks, from the same pool.

This is why the startup log reports one number, max_total_num_tokens, and not two. On our Mac with --mem-fraction-static 0.4, that number was 181,574 tokens. The model allows a context of 40,960 tokens. So the pool holds four full-length conversations, or about 180 conversations of 1,000 tokens each. Running requests and cached prefixes compete for those same slots.

Our position: count tokens, not gigabytes

Size a deployment from max_total_num_tokens, not from free memory. Multiply your typical context length by the concurrency you need, and compare the product with that number. If the product is larger, the scheduler retracts or queues requests, and section 2.3 shows what that costs.

The two levers that change the number are the memory fraction in section 3.2 and the KV data type. --kv-cache-dtype defaults to auto, which keeps the model data type unless the checkpoint asks for FP8 code. FP8 halves the bytes per token.

A field report shows the size of that lever. Cloudflare reported that FP8 KV raised the context held in memory for Kimi K2.6 from about 686,000 to about 1.37 million tokens field Cloudflare 2026-08. The same report measured a few percent less throughput per request at low concurrency. At concurrency 64, the BF16 configuration ran out of memory, and the FP8 configuration reached 2,192 tokens per second. You trade a little speed per token for twice the users. Read the Cloudflare post for their accuracy checks before you copy the change.

What to measure

• Read bytes_per_slot or compute it, and check that the two agree.
• Read max_total_num_tokens in the startup log, and divide it by your typical context length.
• Watch token usage in the scheduler log. It is the fraction of the pool that running requests and cached prefixes hold.

The lesson on inference optimization builds a KV cache from scratch, if you want the mechanism below SGLang.

Sources:python/sglang/srt/model_executor/pool_configurator.py (_compute_cell_size, calculate_pool_sizes), python/sglang/srt/mem_cache/memory_pool.py (ReqToTokenPool), python/sglang/srt/mem_cache/allocator/token.py (TokenToKVPoolAllocator), python/sglang/srt/hardware_backend/mlx/model_runner.py (_compute_pool_size), python/sglang/srt/arg_groups/fields/model.py (kv_cache_dtype) at v0.5.22, capture/out/server-default.log, the SGLang paper (arXiv 2312.07104), the Cloudflare report of 2026-08-03.

Our first SGLang server on the capture Mac ran for four minutes. Then a benchmark at 32 concurrent requests crashed it with a Metal out-of-memory error, and every process of the server exited. The KV pool at that moment was 95 percent empty. The memory had gone to the pool, and the next prefill had nowhere to run.

When you finish this section, you can read how SGLang divides memory at startup and predict when that division fails. You can also set --mem-fraction-static on purpose.

The crash we recorded

We started the server twice before the capture kit existed, with the same command and no --mem-fraction-static. The two startups sized the pool very differently.

# run 1 startup
[2026-10-16 13:30:31] Auto-sized attention KV pool: sys_available=27.81 GB, mlx_limit=51.8 GB, mlx_used=0.31 GB, kv_budget=24.25 GB, bytes_per_slot=114688, pool_size=227022
# run 2 startup
'mem_fraction_static': 0.872
[2026-10-16 13:38:58] Auto-sized attention KV pool: sys_available=38.29 GB, mlx_limit=51.8 GB, mlx_used=0.31 GB, kv_budget=33.39 GB, bytes_per_slot=114688, pool_size=312581

Nothing changed in SGLang between the runs. Other programs on the Mac used less memory at the second start, so sys_available grew from 27.81 GB to 38.29 GB. The pool grew with it, from 24.25 GB to 33.39 GB. Both budgets are 87.2 percent of the free memory, because the automatic mem_fraction_static resolved to 0.872.

[2026-10-16 13:42:53] Decode batch, #running-req: 32, #token: 16095, token usage: 0.05, cuda graph: False, gen throughput (token/s): 564.55, #queue-req: 0
[2026-10-16 13:42:55] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 9, #pending-token: 0, cuda graph: False, input throughput (token/s): 66.85
RuntimeError: [METAL] Command buffer execution failed: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory).

[2026-10-16 13:42:57] SIGQUIT received. signum=None, frame=None. It usually means one child failed.
…
[2026-10-16 13:43:02] kill_process_tree called: parent_pid=81559, include_parent=True, pid=81559

Read token usage: 0.05 first. Thirty-two requests held 16,095 tokens, which is 5 percent of a 312,581-token pool. The pool reserved 34 GB and used under 2 GB of it. The activations of the next forward pass needed memory outside the pool, and the operating system had none left. One failed command buffer took down the scheduler, and the server killed its whole process tree.

How SGLang decides the split

--mem-fraction-static: the share of memory that SGLang gives to the model weights plus the KV pool. Everything outside that share is for activations, CUDA graphs, and other programs. The flag defaults to None, and SGLang computes a value at startup code.

On a GPU, the automatic value comes from a reserve. SGLang starts with 512 MB and adds 1.5 MB for each token of the prefill chunk code. Then it adds more for tensor and pipeline parallelism and for CUDA graphs. On a GPU with more than 60 GB, the reserve is at least 10 GiB. The fraction is then the GPU memory minus the reserve, divided by the GPU memory. When SGLang cannot read the device capacity, it uses 0.95.

After the weights load, SGLang measures free memory again. The KV budget is the free memory minus the 1 - fraction share of the memory that was free before loading code. The pool holds that budget divided by the bytes per token from section 3.1. If the budget is not positive, startup stops and prints the smallest fraction that would work.

The MLX backend on Apple silicon uses a simpler rule code. It takes the smaller of two numbers. The first is the fraction of the Metal working set, minus memory already in use. The second is the fraction of system memory available right now. The second term is the snapshot that changed between our two runs.

One public tuning write-up describes the fraction the other way round, as weights plus buffers, with the KV cache in the remainder. The code says the opposite: the static share holds the weights and the KV pool.

The default assumes the machine belongs to SGLang

The automatic rule is reasonable on a dedicated GPU. Nothing else allocates memory there, and the reserve covers the activations of one prefill chunk and the CUDA graphs. The rule fails in three common cases.

• A shared machine. A laptop, a workstation, or a node with other jobs. The snapshot at startup does not hold for the life of the server.
• Unknown capacity. On a device where SGLang cannot read the memory size, the fraction becomes 0.95 and leaves 5 percent for activations. Issue #39675 reports exactly that failure on Apple MPS, an out-of-memory error on the first prefill issue #39675.
• Extra memory the estimate does not know about. Issue #42510 shows draft-model copies that stay in memory while the target pool is sized, so the draft pool allocation fails issue #42510. Issue #6027 shows a 64K-token DeepSeek-V3 prompt that ran out of memory in attention, although the log reported room for 274,934 KV tokens issue #6027.

Our position: set the fraction yourself on any shared machine

On a machine that runs anything besides SGLang, pass --mem-fraction-static yourself. The capture kit uses 0.4 on the Mac. With that value, the same benchmark that crashed run 2 finished all 128 requests at concurrency 32.

[2026-10-16 13:58:03] Auto-sized attention KV pool: sys_available=48.49 GB, mlx_limit=51.8 GB, mlx_used=0.31 GB, kv_budget=19.39 GB, bytes_per_slot=114688, pool_size=181574

The budget is now 19.39 GB, which is 0.4 of the 48.49 GB snapshot. The pool still holds 181,574 tokens, far more than 32 requests of 640 tokens need. A smaller pool cost nothing in this workload, because the workload never filled the pool.

On a dedicated GPU, trust the automatic value first. Raise the fraction in steps of 0.02 only when the scheduler log shows a full pool and retractions. Find the upper limit in staging with your longest prompts, then run production one or two steps below it. Lower it when long prompts fail in attention, as in issue #6027.

The fraction also changes comparisons. One independent benchmark ran SGLang with its automatic fraction and vLLM with its default of 0.9 of GPU memory field Rawlinson 2025-12. The two engines had KV pools of very different sizes. Read section 5.3 before you trust a comparison that does not state both values.

What to measure

• The max_total_num_tokens line at startup. It is the result of the whole calculation.
• token usage in decode lines. A value near 1.0 means the pool is full.
• The free memory of the machine during the heaviest prefill, from the operating system.

One more caution for Apple silicon. The kv_cache_memory_usage_gb metric read 0.0 in our capture, because the MLX backend allocates its own pool outside the accounting that the metric reads. Use the MlxAttentionKVPool log line, which reported about 19,860 MB.

Sources:python/sglang/srt/arg_groups/memory_hook.py (handle_gpu_memory_settings), python/sglang/srt/mem_cache/kv_cache_configurator.py (_profile_available_bytes), python/sglang/srt/hardware_backend/mlx/model_runner.py (_compute_pool_size), python/sglang/srt/arg_groups/fields/schedule.py (mem_fraction_static) at v0.5.22, capture/out/oom-exploratory.log, capture/out/server-default.log, capture/out/metrics.txt, issues #39675, #42510, and #6027, the Rawlinson benchmark of 2025-12-05.

A chat product sends the same 2,000-token system prompt with every question. A server without prefix reuse computes those 2,000 tokens for every request, and every user waits for the same work again. On our Mac, that repeated work made a shared-prefix benchmark take 129.7 seconds. With the radix cache on, the same benchmark took 48.8 seconds.

When you finish this section, you can explain how the tree finds a prefix and why a running request keeps its prefix. You can also say how eviction removes cached tokens. You can also read the scheduler log lines that prove a hit.

The same workload with and without the tree

The capture kit runs SGLang's own shared-prefix benchmark twice. Four groups of 16 prompts share a 2,048-token system prompt per group, and each prompt adds a 128-token question. The second server starts with --disable-radix-cache.

We measured a 2.7 times shorter run and a 5.3 times shorter median wait for the first token. The scheduler log shows where the time went.

[2026-10-16 14:01:00] Prefill batch, #new-seq: 7, #new-token: 955, #cached-token: 15102, token usage: 0.05, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 1472.94

[2026-10-16 14:02:34] Prefill batch, #new-seq: 3, #new-token: 4096, #cached-token: 0, token usage: 0.06, #running-req: 2, #queue-req: 3, #pending-token: 7883, cuda graph: False, input throughput (token/s): 2261.21
[2026-10-16 14:02:36] Prefill batch, #new-seq: 3, #new-token: 4096, #cached-token: 0, token usage: 0.08, #running-req: 4, #queue-req: 1, #pending-token: 3787, cuda graph: False, input throughput (token/s): 2245.89
[2026-10-16 14:02:38] Prefill batch, #new-seq: 2, #new-token: 3787, #cached-token: 0, token usage: 0.10, #running-req: 6, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 1626.02

With the tree, seven new requests needed 955 new tokens, and 15,102 tokens came from the cache. Without the tree, the scheduler filled three batches of up to 4,096 tokens for eight requests, all computed from zero. The 4,096 is the chunked prefill size from section 2.2.

The tree

RadixCache: a radix tree over token sequences, whose nodes hold the KV slot indices of the tokens on their edges. The design comes from the SGLang paper, which calls the mechanism RadixAttention paper. A radix tree stores runs of tokens on its edges, so one node can hold a whole system prompt.

TreeNode: one node of the tree. It holds its children, keyed by their first token, and the token run on its edge code. It also holds the KV slot indices for those tokens, a lock count, a last-access time, and a hit count.

Four operations do all the work.

1. match_prefix walks from the root and follows the child whose first token matches. If the match ends inside an edge, it splits that node so the boundary is exact. Every node on the path gets a new access time.
2. insert walks the same way and adds a leaf for the tokens that did not match. If another request already inserted the same tokens, SGLang frees the duplicate slots and points the request at the tree's copy.
3. inc_lock_ref and dec_lock_ref count the running requests that use a node. A count above zero protects the whole path to the root.
4. evict frees leaves that no request locks, oldest first, and moves up the branch as leaves disappear.

One rule surprises people. A match can cover at most the prompt length minus one code. The last prompt token always goes through prefill, because the model needs its logits to sample the first output token. Our cache experiment shows that rule: a 2,999-token prompt reported 2,998 cached tokens and 1 new token.

Which tree you get at v0.5.22

The class name in old blog posts and issues is RadixCache. At v0.5.22 the default is UnifiedRadixCache, one tree that also handles sliding-window and Mamba state code. The tree walks run in a Rust core by default since v0.5.21. The log on our Mac shows the exception.

[2026-10-16 13:58:06] Using the Python TreeCore: the Rust TreeCore supports Linux only
[2026-10-16 13:58:06] Init Unified Radix Cache. Components: (<ComponentType.FULL: 0>,). Tree Core: UnifiedTreeCore
[2026-10-16 13:58:06] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=False hybrid_ssm=False hicache_attached=False

The no-cache server printed the same three lines. --disable-radix-cache still builds UnifiedRadixCache, in a disabled mode that matches nothing and inserts nothing. The separate ChunkCache class that older posts describe was deleted on 2026-10-08 issue #43201.

How the scheduler uses the tree

The default --schedule-policy is fcfs, first come, first served code. The paper's longest-prefix-match order, lpm, is a choice you make. The code sorts by prefix length only while 128 or fewer requests wait, because matching every waiting request costs time code. Above that, lpm uses the FCFS order.

The scheduler also avoids a waste inside one round. When several waiting requests share a prefix that is not cached yet, it lets one of them prefill the shared part first. The others reuse it in the next round. Section 2.2 covers the policies in full.

Our position: leave the radix cache on

The cache costs almost nothing when prompts do not repeat. Cached tokens live in the same pool as running requests, and eviction frees them on demand. We know of one reason to turn it off: to test whether a wrong answer comes from reused KV. Even then, try --kv-canary log first, which checks KV slots for corruption while the cache stays on code.

Then measure the hit rate together with every other feature you add. Issue #32459 is the warning issue #32459. A team that replayed multi-turn agent traffic saw a deep-prompt hit rate of 97 percent without speculative decoding, and 40 to 53 percent with EAGLE. Input throughput fell from 43,981 to 12,881 tokens per second, with no error in the logs. The candidate cause was the extra KV that speculative decoding reserves, which forced the tree to evict cached prompts.

Field reports put the folklore in proportion. vLLM also reuses prefixes, so engine comparisons measure two caches against each other. One independent test of 16 models on an A10G found SGLang ahead on repeated prefixes field Varad More 2026-04. The gain was 20 to 40 ms of first-token time at 7B to 9B. A RunPod test of a 70B model with a cached 7,000-token context measured 35.0 against 32.8 tokens per second per stream field RunPod 2025-06. The SGLang paper reports up to 6.4 times higher throughput paper, which is the authors' best case against the systems of 2024.

Sources:python/sglang/srt/mem_cache/radix_cache.py (TreeNode, match_prefix, insert, evict, inc_lock_ref), python/sglang/srt/mem_cache/unified_radix_cache.py, python/sglang/srt/mem_cache/unified_cache/tree_core_registry.py, python/sglang/srt/mem_cache/registry.py, python/sglang/srt/managers/schedule_batch.py (_compute_max_prefix_len), python/sglang/srt/managers/schedule_policy.py, python/sglang/srt/arg_groups/fields/schedule.py at v0.5.22, capture/out/bench-shared-prefix.txt, capture/out/no-radix-cache-bench-shared-prefix.txt, capture/out/server-default.log, capture/out/server-no-radix-cache.log, capture/out/cache-experiment.txt, the SGLang paper (arXiv 2312.07104), issues #32459 and #43201, the RunPod report of 2025-06-15, the Varad More benchmark of 2026-04-22.

A team runs DeepSeek-V4 for long agent sessions on eight GPUs, with hierarchical caching turned on. Their dashboards show #cached-token: 0 on requests that repeat 100,000 tokens of history. They tune the cache tiers and the write policy. The hit rate barely moves. Then they find a random hash that their relay put into the system prompt of every request.

When you finish this section, you can predict which prompts hit the prefix cache before you send them. You can also prove the hit rate with cached_tokens, the scheduler log, and /metrics.

The experiment: one identifier, three places

The capture kit sends one 2,999-token system prompt and one short question, five times in each of three forms. The server runs with --enable-cache-report, so each response reports its cached tokens to the client. The kit sends two warm-up requests first.

• Stable: the same system prompt every time.
• ID at start: the system prompt starts with Request id <random hex>., a new value on every request.
• ID at end: the same random ID goes at the end of the system prompt.

…
{"summary": "stable-system-prompt", "median_ttft_ms": 45.1, "min_ttft_ms": 34.0, "max_ttft_ms": 62.5, "median_cached_tokens": 2998.0, "prompt_tokens": 2999}
{"summary": "id-at-start", "median_ttft_ms": 1255.3, "min_ttft_ms": 1102.6, "max_ttft_ms": 2197.3, "median_cached_tokens": 5, "prompt_tokens": 3034}
{"summary": "id-at-end", "median_ttft_ms": 88.2, "min_ttft_ms": 83.7, "max_ttft_ms": 118.1, "median_cached_tokens": 2985, "prompt_tokens": 3028}

We measured a 28 times longer wait for the first token when the ID sits at the start: 1,255 ms against 45 ms. The same ID at the end cost 43 ms more than the stable prompt. The tokens are almost the same in all three cases. Only their order differs.

The cached-token counts explain the times. The stable prompt reused 2,998 of 2,999 tokens, the maximum from section 3.3. The ID at the start matched only the first 5 tokens, the ones that come before the random hex digits. Every token after a change is a different branch of the tree, even when the rest of the prompt is identical.

Why the order matters

The radix tree compares tokens from the first one onward, and it stops at the first difference. Nothing after a difference can match, because the attention values of every later token depend on every earlier token. A cache that reused the later tokens would return wrong answers. So prefix caching is exact by design, and it is unforgiving by the same design.

The same rule applies to each part of a chat request. The chat template turns the messages into one token sequence in a fixed order: system prompt, tool list, earlier turns, new message. A change in the system prompt invalidates everything. A change in the newest message invalidates only that message.

The incident

Issue #35129 is the story at the top of this section issue #35129. The reporter found three causes stacked on top of each other.

1. The relay put a per-session random cch= hash and a rotating system message into each request. Removing them moved the engine hit rate from 66.8 percent to 94.4 percent.
2. The client wrote its task-list snapshot at the end of the system prompt. Every task update changed the system prompt, so it invalidated more than 100,000 tokens of history after it.
3. Hierarchical cache load-back, when it could not find room, evicted the prefixes of other sessions. The reporter counted 250,000 evict calls against 376 completed load-backs.

The first fix was a prompt edit. No server flag in section 3.5 could have recovered the 27.6 points that the hash cost.

Our position: fix prompts before you tune the server

Treat the prompt layout as part of the serving system. Before you change any cache flag, check these rules against your real requests.

• Put content that never changes first: the system prompt, then the tool list in a fixed order.
• Put content that changes per session after that, and content that changes per request last.
• Never put timestamps, request IDs, session hashes, or random sampling seeds in the system prompt. Send them as request metadata, or at the end.
• Keep tool and function definitions in a stable order. A JSON object whose keys change order is a different token sequence.
• Append to the conversation. Do not rewrite earlier turns, for example to summarize them, unless you accept a full prefill.

Two server-side settings break reuse in ways that look like prompt problems. A request for prompt log probabilities from position k can reuse only the first k tokens code. Different LoRA adapters never share tree nodes, even for identical tokens, because the adapter is part of the tree key code.

Routing can break reuse too. With several servers, a request that goes to a server without its prefix misses the cache, whatever the prompt looks like. Jarvislabs reported that routing each conversation to the same server moved their hit rate from 64.7 percent to 92.1 percent field Jarvislabs 2026-08. Section 6.4 shows how the gateway routes by prefix.

How to prove a hit

Use three measurements, from the closest to the request to the most general.

1. In the response. Start the server with --enable-cache-report, which defaults to off code. The usage block then reports cached_tokens. The native /generate API returns cached_tokens in meta_info without the flag.
2. In the scheduler log. Every Prefill batch line reports #new-token and #cached-token.
3. In the metrics. /metrics exports sglang:cache_hit_rate, a gauge of recent prefill batches. After the shared-prefix benchmark in section 3.3, our server reported 0.94. The server with --disable-radix-cache reported 0.0.

[2026-10-16 13:58:12] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 2998, token usage: 0.02, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 7.09
…
[2026-10-16 13:58:14] Prefill batch, #new-seq: 1, #new-token: 3031, #cached-token: 3, token usage: 0.02, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 1937.91
…
[2026-10-16 13:58:20] Prefill batch, #new-seq: 1, #new-token: 45, #cached-token: 2983, token usage: 0.02, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 225.19

The lessons on prompt caching and on caching and cost show the same rule for hosted model APIs. Those APIs bill cached input tokens at a lower price.

Sources:python/sglang/srt/mem_cache/radix_cache.py (RadixKey, match_prefix), python/sglang/srt/managers/schedule_batch.py (_compute_max_prefix_len), python/sglang/srt/arg_groups/fields/serving.py (enable_cache_report), python/sglang/srt/entrypoints/openai/serving_chat.py at v0.5.22, capture/out/cache-experiment.txt, capture/out/server-default.log, capture/out/generate.txt, capture/out/metrics.txt, capture/out/no-radix-cache-metrics.txt, issue #35129, the Jarvislabs report of 2026-08-26.

A coding agent holds a session of eight turns, and its history grows past 25,000 tokens. The GPU pool holds a few dozen such sessions. The ninety-first user arrives, and SGLang must evict a cached prefix to make room. When the first user returns, their history is gone, and the server computes all 25,000 tokens again.

When you finish this section, you can choose an eviction policy from a measured hit rate. You can also decide when hierarchical caching pays for its cost and which of its settings to change first.

What eviction removes

Eviction removes only cached tokens that no running request locks, the leaves of the radix tree. It never removes the KV of a running request. That case is retraction, and section 2.3 covers it. Eviction starts from the leaves with the lowest score and moves toward the root.

--radix-eviction-policy decides the score code. The default is lru.

Tail-Optimized LRU (tlru): a policy that keeps only the part of a conversation that the next turn's latency target needs, and evicts the rest first. It comes from a paper that the code names, arXiv 2510.15152. The SGLang team reports 43.9 percent lower p99 inter-token latency on DeepSeek-V4-Pro at concurrency 32 team 2026-10. That number comes from the release notes of v0.5.21, and we found no independent test of it.

Our position: keep lru until you have measured a reason

The SGLang docs give the same advice: start with lru and change it only against a measured hit rate docs. Frequency-based policies hurt when popularity shifts, because yesterday's popular prompt stays in memory and today's prompt pays for it. Change the policy only when you can name the workload that the new policy serves, and measure the hit rate before and after.

HiCache: three tiers for one tree

HiCache: SGLang's hierarchical cache, which extends the radix tree from GPU memory to host memory and to a storage backend. --enable-hierarchical-cache turns it on, and it defaults to off code. The tiers have names in the code and the docs.

• L1: the GPU pool from section 3.1.
• L2: a pool in pinned host memory, private to one server instance.
• L3: an optional storage backend shared across instances, set with --hicache-storage-backend. The choices include file, mooncake, hf3fs, and nixl.

The tree records where the KV of each node lives. A match returns a GPU hit and a host hit length. SGLang then loads the host part back into fresh GPU slots, one layer ahead of the computation that needs it docs.

--hicache-write-policy decides when a node gets a copy in host memory. The default is write_through. The code sets a hit threshold of 1 for write_through and 2 for write_through_selective code. So write_through copies a prefix on its first hit, and write_through_selective waits for a second hit. write_back copies a node only when the GPU evicts it.

The settings that decide whether it helps

L3 prefetch has its own rules. SGLang skips a storage prefetch when the storage hit is shorter than 256 tokens code. The default prefetch policy is timeout, which stops waiting for storage after a time that grows with the prompt.

The docs and the code disagree on that time. The docs give a base of 2 seconds, 0.1 seconds per 1,024 tokens, and a maximum of 30 seconds docs. The code at v0.5.22 uses a base of 1.0 second and 0.25 seconds per 1,024 tokens code. We found no maximum in the cache code. The code wins. A 100,000-token prompt can wait about 25 seconds for storage before SGLang computes it again.

The host pool size comes from --hicache-ratio or --hicache-size. Without either, SGLang takes up to --hicache-host-memory-fraction, which defaults to 0.8 of available host memory code.

What other teams measured

We could not run HiCache on the capture Mac, and the field evidence is thin. The numbers we found come from the SGLang team and its partners.

• The SGLang RBG team and the Mooncake team ran a multi-turn benchmark on Kubernetes field Alibaba Cloud 2026-03. The average time to first token was 5.91 seconds with GPU memory only. The L2 tier lowered it to 3.77 seconds, and a Mooncake L3 lowered it to 2.58 seconds.
• The SGLang HiCache post quotes Novita team 2025-09. Novita reported a 56 percent lower average time to first token, and a hit rate that rose from 40 to 80 percent.

The failure reports are better documented than the successes. In issue #35129, load-back that found no room evicted other sessions' GPU prefixes, with 250,000 evict calls against 376 completed load-backs issue #35129. Issue #30760 shows one tensor-parallel rank waiting forever in a prefetch check while the other ranks moved on issue #30760. Issue #42465 shows bursts of 540,000-token prompts that hang every rank with HiCache on and run without it issue #42465.

Our position: add HiCache for long sessions, and add it in steps

HiCache pays for itself when sessions outlive the GPU pool: multi-turn chat, coding agents, and long documents asked several times. For short one-time prompts it adds copies and load-backs with nothing to reuse. Add it in this order, and measure the hit rate and the p99 time to first token at each step.

1. Make prompts prefix-stable first, as in section 3.4.
2. Turn on L2 only, with the default write_through.
3. Switch to write_through_selective if host copies use too much PCIe bandwidth.
4. Add an L3 backend when several instances should share prefixes.
5. Change the eviction policy last.

Sources:python/sglang/srt/mem_cache/evict_policy.py, python/sglang/srt/mem_cache/unified_radix_cache.py (write_through_threshold, prefetch_timeout_base), python/sglang/srt/mem_cache/hybrid_cache/hybrid_pool_assembler.py (prefetch_threshold), python/sglang/srt/arg_groups/fields/memory.py at v0.5.22, docs/docs/advanced_features/hicache_design.mdx and radix_eviction_policy.mdx at v0.5.22, the v0.5.21 release notes, the SGLang HiCache post of 2025-09-10, the RBG and Mooncake post on the Alibaba Cloud blog of 2026-03-10, issues #35129, #30760, and #42465, arXiv 2510.15152.

A team serves a hybrid model that mixes linear-attention layers with full-attention layers. They turn on HiCache and run twenty requests that hit the host tier. All twenty answers are fluent and on topic for a different prompt. The usage block reports full cached tokens, and nothing in the logs warns them.

When you finish this section, you can say why hybrid models break the simple picture of the radix tree. You can also test the cache on your model before your users test it for you.

Three kinds of layer state

The radix tree in section 3.3 works because full attention stores one key and one value per token. A cached prefix of any length is valid, so the tree can split a node at any token. Two other kinds of layer break that rule.

Sliding-window attention (SWA): attention that reads only the last few thousand tokens. A SWA layer needs only the window, so SGLang keeps a smaller pool for SWA layers and drops KV that leaves the window. A cached prefix is useful only where the window's KV still exists.

Mamba layer: a state-space or linear-attention layer that keeps one fixed-size recurrent state per sequence, updated token by token. The state at token 2,000 summarizes all 2,000 tokens. It cannot be cut back to token 1,500. A prefix can resume only where SGLang saved a snapshot of the state.

Our capture model has neither. The startup log says so.

[2026-10-16 13:58:06] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=False hybrid_ssm=False hicache_attached=False
…
[2026-10-16 13:58:08] Attention KV pool initialized: pool_size=181574 (buffer size 181575 incl. padding slot 0), 28 full-attention layers (0 sliding-window layers stay per-request), 8 kv_heads, 128 head_dim

Read these two lines on your own model first. hybrid_swa=True or hybrid_ssm=True means that everything in this section applies to you.

How SGLang handles each kind

UnifiedRadixCache keeps one tree with a component for each kind of state code. Each node holds data for the full-attention component and, when needed, for the SWA and Mamba components. A match advances only while every component agrees that the state at that position is valid.

For SWA, the tree keeps a marker where window KV was dropped, and a match stops at the marker. --swa-full-tokens-ratio sets the size of the SWA pool against the full pool. It defaults to None, and model-specific overrides set it. The DeepSeek-V4 override sets it to 0.1 code.

For Mamba, the tree stores state snapshots on some nodes and copies a snapshot when a request reuses it. --mamba-radix-cache-strategy chooses how SGLang keeps snapshots: auto by default, no_buffer, extra_buffer, or extra_buffer_lazy code. --mamba-track-interval sets how often decode saves a snapshot, every 256 tokens by default.

What goes wrong, in the issues

Most open cache bugs at v0.5.22 involve hybrid models. These four show the range.

1. A wrong answer from a host-tier hit. Issue #39830 is the opening story of this section issue #39830. A hybrid linear-attention model with HiCache gave fluent answers to a different prompt on 20 of 20 host-tier hits. Cold requests and GPU hits were correct. Earlier reports of the same class exist on the GPU path.
2. Zero reuse for an identical prompt. In issue #22935, the match of a repeated prompt ends inside an edge, and the node split creates a node with no Mamba state issue #22935. The match stops at the last node with state, so the identical prompt reuses nothing.
3. A strategy that makes the first token 4.7 times slower. In issue #37834, an option forced the no_buffer strategy issue #37834. The snapshot sat at a depth that later requests could not reach. On a retrieval workload the hit rate fell from 88.2 to 65.9 percent. The mean time to first token rose from 226 ms to 1,063 ms.
4. A default that slowed decode. Issue #36131 reports a new default that frees SWA slots outside the window issue #36131. It reduced single-stream decode throughput for hybrid SWA models on DGX Spark and Thor.

Our position: test the cache on a hybrid model before you trust it

On a full-attention model, a wrong cache hit is rare enough that we leave the cache on and move on. On a hybrid model, run three checks before production. Each one takes minutes.

1. Correctness. Send the same prompts cold, with a GPU hit, and, if you use HiCache, with a host-tier hit. Compare the outputs at temperature 0. Run --kv-canary log during the test, which checks KV slots for corruption and keeps the server running code.
2. Reuse. Repeat a prompt and read #cached-token. An identical prompt that shows a small count has hit the split problem of issue #22935.
3. Latency. Measure the time to first token with --mamba-radix-cache-strategy auto and with the strategy your deployment guide recommends. Keep the faster one only if the hit rate holds.

Then read the open issues for your model family before each upgrade. Between 2026-09-01 and the v0.5.22 tag, 334 commits touched the cache code. On hybrid models, a new release can fix one of these bugs and add another.

Sources:python/sglang/srt/mem_cache/unified_radix_cache.py, python/sglang/srt/mem_cache/unified_cache/components/README.md, python/sglang/srt/arg_groups/fields/exec_.py (mamba_radix_cache_strategy, mamba_track_interval), python/sglang/srt/arg_groups/fields/schedule.py (swa_full_tokens_ratio), python/sglang/srt/arg_groups/model_overrides/deepseek_v4.py, python/sglang/srt/arg_groups/fields/observability.py (kv_canary) at v0.5.22, capture/out/server-default.log, issues #39830, #22935, #37834, and #36131.

You serve a small model to one user on an H100 and watch tokens arrive. The GPU is fast, yet a profiler shows gaps between kernels. A field report from Hazy Research field Hazy Research 2025-09 measured that SGLang and vLLM used about half of the H100 memory bandwidth for Llama-1B at batch 1. The GPU waits for the CPU to tell it what to do next.

When you finish this section, you can name the mode of any forward pass and explain what a CUDA graph saves. You can also read the batch sizes SGLang captures and keep busy decode inside a graph.

Every forward pass has a mode

ForwardMode: the label on a batch that tells the model runner what kind of pass to run. The scheduler sets it, and the model runner, the attention backend, and the graph runners all read it.

SGLang calls any pass that writes many KV entries per request "extend". That is why TARGET_VERIFY counts as extend in the code. The target model runs over several draft positions per request, like a small prefill with a tree mask.

The cost: one decode step is a long list of small launches

A transformer layer is a sequence of kernels: a norm, the QKV projection, attention, the output projection, a second norm, and the MLP. A 28-layer model launches hundreds of kernels to produce one token. During prefill each kernel works on thousands of tokens, so launch time is small next to compute time. During decode each kernel works on one token per request.

At batch 1 the GPU often finishes a kernel before the Python process has sent the next one. The obvious fix is a faster launch path, such as fewer Python calls. That helps a little, but each kernel still needs its own CPU launch, and the count does not change.

CUDA graph: a record of kernel launches with fixed tensor shapes and addresses, which the GPU can replay with one CPU call.

The record has a price. Every captured shape uses GPU memory, and a batch must match a captured shape exactly. SGLang pays that price for decode and pads each batch up to the next captured size.

What SGLang captures, and when it gives up

Capture bucket: one batch size that SGLang records a decode graph for at startup. Without speculative decoding, v0.5.22 captures sizes 1, 2, 4, 8, and 12, then every eighth size from 16 to 256. Above 256 it captures every sixteenth size to 511, then every thirty-second size up to the largest. A batch of 13 requests replays the graph for 16 and wastes three slots.

The largest bucket, max_bs, depends on GPU memory and tensor parallel size. The startup code picks it when you do not set --cuda-graph-max-bs-decode.

When a decode batch is larger than max_bs, the graph runner cannot replay. The step runs eager, with one launch per kernel. Nothing fails and no warning appears. The scheduler log line for that batch says cuda graph: False, and latency per token rises.

The code does not change your shapes for prefill the same way. Since PR #29458 (2026-07-01), CUDA builds use a breakable graph for prefill. Breakable graph: a capture that ends a segment at each op marked to run eager, runs that op eagerly, and starts the next segment. A model with one op that a graph cannot hold still gets graphs for every other op. On other platforms prefill graphs are off, and our Mac capture logs both facts.

… Disable prefill CUDA graph because cuda_graph_config resolved prefill.backend='disabled' (e.g. via --cuda-graph-backend-prefill=disabled or auto-disable rules).
… Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, …

Every scheduler batch line ends with cuda graph:. On a CUDA server, watch that field on your decode batches.

The older torch.compile path for prefill graphs is being removed in PRs #42285 to #42287. A blog post that says "piecewise CUDA graphs use torch.compile" describes that older path.

The cliff in field reports

A field report from Jarvislabs field Jarvislabs 2026-05 ran Qwen2.5-7B on one H100 with default SGLang flags. At concurrency 300, mean inter-token latency was 17.72 ms against 12.28 ms for vLLM. With no concurrency cap it was 100.92 ms against 28.84 ms. The post does not record the SGLang version or the server logs.

Our reading is a hypothesis, labeled as one. The default decode max_bs on that GPU is 256, and the latency break sits just above it. The maintainers saw the same shape on smaller cards. The code comment on the 20 to 35 GB tier says the old cap "forced eager decode at bs>=32 and collapsed high-concurrency throughput vs vLLM" code. PR #37898 raised that cap from 24 to 48 on 2026-09-04.

Our position

Keep CUDA graphs on, and size them to your traffic. The default max_bs suits moderate concurrency. If your benchmark or your production peak runs more requests than that, set --cuda-graph-max-bs-decode and --max-running-requests to your target before you measure. SemiAnalysis InferenceX recipes field InferenceX 2026-05 derive both values from the concurrency they test.

Do not turn graphs off to fix memory. If capture runs out of memory, lower --cuda-graph-max-bs-decode or --mem-fraction-static. A server without decode graphs pays launch time on every token for every user.

Two flag names changed in v0.5.22, and old recipes break on them. --disable-cuda-graph still works, and the help text marks it as deprecated.

  --disable-cuda-graph  Deprecated. Use --cuda-graph-
                        backend-{decode,prefill}=disabled instead.

The flag --cuda-graph-max-bs no longer exists. We passed it to the v0.5.22 argument parser and it exited with ambiguous option: --cuda-graph-max-bs could match --cuda-graph-max-bs-decode, --cuda-graph-max-bs-prefill.

Incidents

Capture runs out of memory with data-parallel attention. Issue #23769 (2026-04-26): DeepSeek-V4-Pro-FP8 on two nodes of eight H200 exited during graph capture. With data-parallel attention, the graph batch size counts per rank, so the captured graphs were larger than the traffic needed. The fix was a smaller graph batch size or a lower --mem-fraction-static.

A renamed field broke the Python API. Issue #36478 (2026-08-26): sgl.Engine(..., disable_piecewise_cuda_graph=True) raised TypeError, while the command-line flag still worked. PR #23906 folded the field into the per-phase graph config and kept only the command-line alias. The fix is cuda_graph_backend_prefill="disabled".

A new GPU needed eager mode. A field report from joshua8.ai field joshua8 2026-01 ran SGLang on an RTX PRO 6000 Blackwell (SM120). It needed --disable-cuda-graph to run at all. On new architectures, check that capture works before you compare engines.

Sources:capture/out/server-default.log, capture/out/serve-help.txt, python/sglang/srt/model_executor/forward_batch_info.py (ForwardMode), model_executor/model_runner.py (_forward_raw), model_executor/cuda_graph_config.py (default_prefill_backend), model_executor/runner/decode_cuda_graph_runner.py (can_run_graph), arg_groups/memory_hook.py (_handle_gpu_memory_settings), arg_groups/cuda_graph_hook.py (generate_decode_cuda_graph_batch_sizes), and server_args.py (deprecated flags) at v0.5.22; the argument parser of the installed v0.5.22 package; PRs #23906, #29458, #37898, #42285; issues #23769 and #36478; field reports by Hazy Research (2025-09-28), Jarvislabs (2026-05-18), joshua8.ai (2026-01-10), and SemiAnalysis InferenceX (2026-05-26).

In January 2026 a user upgraded FlashInfer and served Qwen3-235B on H20 GPUs. Median inter-token latency went from 39.44 ms to 283.52 ms, seven times slower, and nothing else in the setup had changed. The FA3 kernel gave 56.73 ms on the same run. That report is issue #17411, and the code that picks your attention backend at v0.5.22 still cites it.

When you finish this section, you can name the backend SGLang picks for your GPU and model, and explain why. You can also split prefill and decode across two backends and decide when to override the default.

What an attention backend is

Attention backend: the class that computes attention over the paged KV cache for one batch. Every backend implements the same interface. It builds per-batch metadata such as page tables and index buffers. It prepares state for CUDA graph capture and replay. It runs forward_extend for prefill and forward_decode for decode.

The model code never calls a kernel by name. Each attention layer calls whatever backend the model runner installed at startup. So one flag swaps the kernel for every layer of every model.

At v0.5.22 the flag accepts 25 names. Two are aliases (nsa for dsa, and compressed for dsv4), so there are 23 backends.

The obvious fix, and why it fails

The obvious fix is to read a benchmark, find the fastest kernel, and pass it on every server. That fails because each backend has limits that a benchmark does not show.

trtllm_mha rejects speculative decoding with topk above 1 and does not run on SM120. FlashInfer cannot run models that use attention sinks. fa4 has a native page size of 128. Some backends accept FP8 KV caches and some do not. A kernel that wins on one model and GPU can refuse to start on the next.

How SGLang picks a default

When you do not pass --attention-backend, startup calls one function that reads your GPU, the attention type of the model, and your speculative decoding settings.

For MHA models, such as Llama and Qwen, the order is this:

1. On Hopper with CUDA 12.3 or later, and no speculative topk above 1, the default is fa3. The code comment cites issue #17411.
2. On Blackwell SM100 or SM103, the default is trtllm_mha, or fa4 when K and V widths differ.
3. On AMD the default is aiter, and on Apple MPS it is torch_native.
4. In every other case the default is flashinfer when it is installed and the model has no attention sinks, else triton.

For MLA models, such as DeepSeek V3, Hopper gets fa3 and Blackwell gets flashinfer. AMD gets aiter when each rank has 16 or 128 KV heads. A model override then moves DeepSeek V3 on SM100 to trtllm_mla.

The docstring of the same function says FA3 is avoided when the page size is above 1. The code does not check the page size. When a comment and the code disagree, the code is what runs.

Two backends in one server

Prefill and decode stress different things. Prefill reads long query blocks, and decode reads one query against a long cache. --prefill-attention-backend and --decode-attention-backend set each phase separately. When they differ, SGLang wraps both in a hybrid backend that dispatches by ForwardMode. With speculative decoding, --speculative-attention-mode decides which of the two runs the verify pass. Its default is prefill.

Docs and code disagree

The attention backend page in the docs at v0.5.22 still lists Cutlass MLA and Dual Chunk FlashAttention. PR #32114 deleted both on 2026-09-10, and neither name is a valid choice at the tag. The same page says Blackwell MHA uses trtllm_mha. The code picks fa4 for models with asymmetric K and V widths. In both cases the code wins, and this manual follows the code.

What to measure

Read the resolved backend in the startup log before you benchmark anything. Then change one flag, rerun the same benchmark at your real concurrency, and compare inter-token latency at p50 and p99. Attention is one part of a decode step, so a faster kernel can move end-to-end numbers less than its own benchmark suggests.

On our Mac capture, SGLang ran through the MLX runtime. MLX has its own attention path, so none of these backends ran there.

Our position

Leave --attention-backend unset unless you have a benchmark on your own traffic. The default encodes regressions that other users found, such as #17411, and model overrides, such as DeepSeek V3 on SM100. The same issue is also the exception: a user who measures can beat a default that a library upgrade broke. Measure, then override, and write the reason next to the flag.

Prefer the backend that supports every feature you need over the one that wins a microbenchmark. Speculative decoding with topk above 1 needs flashinfer, fa3, or triton when the page size is above 1.

Incidents

A library upgrade made Hopper seven times slower. Issue #17411 (2026-01-20): FlashInfer 0.6.1 regressed the Hopper attention kernel. Median inter-token latency on H20 went from 39.44 ms to 283.52 ms. SGLang made fa3 the Hopper MHA default two days later.

The other engine lost its graphs. A field report by Varad More field Varad More 2026-04 measured Gemma 3 4B on one A10G at 149 tok/s on SGLang and 84 tok/s on vLLM. The cause was outside attention speed. vLLM ran that model with --enforce-eager, without CUDA graphs, because of its interleaved sliding-window attention. A backend comparison is only fair when both servers replay graphs.

Sources:python/sglang/srt/arg_groups/model_override_base.py (get_default_attn_backend), arg_groups/choices.py (ATTENTION_BACKEND_CHOICES), layers/attention/base_attn_backend.py, layers/attention/attention_registry.py, model_executor/model_runner_components/attention_backend_setup.py, arg_groups/fields/spec.py, and docs/docs/advanced_features/attention_backend.mdx at v0.5.22; PR #32114; issue #17411; field report by Varad More (2026-04-22).

Cloudflare served Kimi K2.6 with a BF16 KV cache and raised concurrency step by step. At 32 concurrent requests the server produced 1,558 tok/s. At 64 it ran out of memory. With an FP8 KV cache the same service reached 2,192 tok/s at 64 requests, about 41% above the best BF16 result field Cloudflare 2026-08. The FP8 cache was slower per token at every concurrency both versions could run. It won because twice as many requests fit.

When you finish this section, you can tell weight quantization from KV cache quantization, and load a quantized checkpoint without breaking detection. You can also size the memory gain before you measure speed, and decide where a lower precision belongs.

Three choices, three flags

Weight quantization: storing the model weights in fewer bits, such as FP8, FP4, or INT4. It shrinks the weights and the memory traffic of each GEMM. --quantization selects the method.

GEMM backend: the kernel library that multiplies quantized weights. --fp8-gemm-backend and --fp4-gemm-backend select it, and both default to auto, which picks by GPU generation.

KV cache dtype: the format of the cached keys and values. --kv-cache-dtype selects it, separately from the weights. Its default auto uses the model dtype.

A model can run FP8 weights with a BF16 KV cache, or BF16 weights with an FP8 KV cache. Each choice moves a different budget, so treat them as three decisions.

Weights: offline first

At v0.5.22, --quantization accepts 31 method names. Most of them describe a checkpoint that someone already quantized: AWQ, GPTQ, ModelOpt FP8 and FP4, compressed-tensors, GGUF, Quark, and others. A few quantize BF16 weights while the model loads: fp8, mxfp8, nvfp4_online, and mlx_q4 or mlx_q8 on Apple silicon.

Offline quantization: the checkpoint already stores low-bit weights and their scales. Online quantization: SGLang converts full-precision weights at load time.

When you load a quantized checkpoint, do not pass --quantization. The model config check reads quantization_config.quant_method from the checkpoint and uses it. If you pass a method that differs, the check accepts only known compatible pairs, such as modelopt_fp4 on a modelopt checkpoint. For a requantization it logs that you should validate accuracy again. Every other mismatch raises at startup.

The flag can also block an upgrade. A GPTQ checkpoint loaded with no flag is detected, converted at runtime, and served by the Marlin kernel. The upgrade code accepts that conversion only when you passed no method or gptq_marlin.

PR #32114 deleted the plain GPTQ CUDA kernel on 2026-09-10, and on a CUDA build the method registry has no gptq entry. The flag parser still accepts --quantization gptq. The model config check then skips the Marlin upgrade and raises Unknown quantization method: gptq. We read this path in the code and did not run it on a GPU.

The KV cache: capacity, with a scale

--kv-cache-dtype accepts auto, bf16, fp8_e4m3, fp8_e5m2, mxfp8 (FA4 only), nvfp4, and fp4_mx_block16. FP8 halves the bytes per token and FP4 quarters them, so the same pool holds two or four times as many tokens. Section 3.1 shows how to compute bytes per token for your model.

FP8 needs a scale for each tensor. The help text of --quantization-param-path says the file "should generally be supplied, when KV cache dtype is FP8" and that without it the scales "default to 1.0, which may cause accuracy issues" code. Run an accuracy check whenever you change the KV cache dtype.

Cloudflare published its check. On Kimi K2.6, BF16 against FP8 KV scored 94.24 and 94.09 on GSM8K, 89.11 and 89.04 on MMLU, and 80.29 and 79.29 on MMLU-Pro field Cloudflare 2026-08. Those are their numbers on their model, and your model can differ.

Quantization is a capacity decision

Cloudflare also tested INT4 weights on GLM 5.2 field Cloudflare 2026-08. Decode got faster, 60 to 92 tok/s at concurrency 1 and 1,672 to 1,933 tok/s at 64. Prefill got slower, from about 10,160 tok/s with FP8 to about 8,660 with INT4.

Decode is limited by memory traffic, so smaller weights help. Prefill is limited by compute, and dequantizing INT4 costs compute. Cloudflare split the two phases across servers and runs INT4 for decode and FP8 for prefill. Section 6.3 explains the split.

Our position

Use an offline-quantized checkpoint and pass no --quantization flag. The docs say the same: "offline quantization is recommended over online quantization" docs. Online quantization adds load time and gives you a checkpoint that nobody evaluated.

Choose a KV cache dtype from memory, not from speed. If your server queues requests because the pool is full, an FP8 cache with a scale file is the cheapest capacity you can buy. If the pool has room, keep the model dtype.

Quantize per phase when you run prefill and decode on separate servers. Smaller weights help decode, and they can slow prefill.

Incidents

The BF16 cache ran out at 64 requests. Cloudflare field Cloudflare 2026-08 reported that Kimi K2.6 with a BF16 KV cache ran out of memory at concurrency 64. The FP8 cache held about 1.37 million tokens against about 686,000 and served the same load.

A container could not run NVFP4. A field report from joshua8.ai field joshua8 2026-01 used an RTX PRO 6000 Blackwell. NVFP4 did not run in its SGLang container, so it compared SGLang on GPTQ-INT4 against vLLM on NVFP4. Compare engines on the same weight format, or the result measures the format.

FP4 needs one GPU generation. A user summary on Pivot News field Pivot News 2026-05 reported that SGLang crashed when RTX 4090 cards joined a Blackwell pipeline. Its FP4 path required compute capability 10.0 and had no software path for older cards. This report reached us through a secondary summary of a Reddit post.

Sources:python/sglang/srt/arg_groups/choices.py (QUANTIZATION_CHOICES), arg_groups/fields/model.py (kv_cache_dtype, quantization_param_path), configs/model_config.py (_verify_quantization), layers/quantization/__init__.py (get_quantization_config), layers/quantization/gptq/gptq.py (override_quantization_method), layers/quantization/fp8.py, and docs/docs/advanced_features/quantization.mdx at v0.5.22; PR #32114; field reports by Cloudflare (2026-08-03), joshua8.ai (2026-01-10), and Pivot News (2026-05-19).

A user turned on EAGLE-3 for Llama-3.1-8B on an A6000 and ran two tests. At batch 1, output went from 42.0 to 92.5 tok/s, more than twice as fast. At concurrency 20, output fell from 217.8 to 164.4 tok/s, a quarter slower. Both numbers are in issue #5274, and both are correct. A maintainer answered that speculative decoding "is for small batch sizes rather than stress tests" issue #5274.

When you finish this section, you can explain draft, verify, and acceptance length, and read the defaults SGLang picks. You can also find the request limit that speculation sets and decide where speculation belongs.

The cost: decode is one token at a time

Each decode step commits one token per request, and the next step needs that token as input. At batch 1 a step reads every weight of the model to compute one token. The GPU has compute to spare and memory traffic is the limit. A larger batch fills that spare compute with other requests.

Speculative decoding fills it with guesses. Draft model: a small model, or a small head trained on the target model, that proposes the next few tokens cheaply. Verify pass: one forward of the target model over all proposed tokens at once. The target accepts the longest prefix that matches what it would have produced, and adds one token of its own.

Acceptance length: the number of tokens committed per verify pass, accepted drafts plus that one bonus token. An acceptance length of 3 means one target forward commits three tokens.

One step in SGLang

1. Draft. The draft model runs --speculative-num-steps times. With --speculative-eagle-topk above 1, each step keeps several candidates per node and grows a tree. A tree builder keeps the best --speculative-num-draft-tokens nodes and builds a tree mask.
2. Verify. The target runs once in TARGET_VERIFY mode over every draft token with the tree mask. For greedy requests it keeps the longest matching path. For sampled requests it compares probabilities with temperature, top-k, and top-p applied.
3. Draft extend. The draft model reads the accepted tokens in DRAFT_EXTEND_V2 mode, so its cache is ready for the next step.

The accept thresholds --speculative-accept-threshold-single and --speculative-accept-threshold-acc default to 1.0. With those values, a greedy request produces the same tokens it would produce without speculation.

The algorithms and the defaults

--speculative-algorithm accepts EAGLE, EAGLE3, NEXTN, STANDALONE, NGRAM, DFLASH, DSPARK, and UNO. NEXTN is another name for EAGLE. For DeepSeek V3 and other models that include multi-token prediction layers in the checkpoint, SGLang sets the draft model path to the model path. NGRAM needs no draft model at all. It looks up repeated token sequences in the text it has already seen.

When you leave the shape unset, SGLang chooses (num_steps, topk, num_draft_tokens) from the model architecture.

With topk 1, the code forces the draft token count to the step count plus one.

The limit nobody asked for

The EAGLE, NGRAM, DFLASH, and DSPARK handlers at v0.5.22 each do the same thing when you did not set --max-running-requests. It sets the value to 48 and logs one warning: "Max running requests is reset to 48 for speculative decoding" code. Speculation needs memory for draft tokens and works best at small batch, so the cap protects the server. It also caps your benchmark.

A throughput test with speculation on and no explicit limit measures a server that admits 48 requests. If your traffic runs 200 concurrent requests, the other 152 wait in the queue, and time to first token grows. The warning prints once at startup, and it scrolls away before the first request.

What to measure

The scheduler logs accept len and accept rate on each decode log line, and /metrics exports sglang:spec_accept_length and sglang:spec_accept_rate. Measure them on your own prompts, because acceptance depends on the text and on the draft. A field report by David Noel Ng field David Noel Ng 2026-08 on two GH200 measured 3.5 to 5.05 tokens on DeepSeek-V4-Flash after a loader fix.

Then sweep concurrency with speculation on and off, and find where the two lines cross. Ant Group field Ant Group 2025-09 measured this on DeepSeek-R1 on H20. Longer drafts gained 21% at batch 1 and lost about 18% at batch 32.

Our position

Treat speculative decoding as a latency tool for low and middle concurrency. Turn it on for interactive traffic with a few users per replica. Leave it off for batch jobs and for servers that run at high concurrency.

Always set --max-running-requests yourself when speculation is on. Choose it from the concurrency where your sweep shows speculation still wins.

Use the draft layers that come with your model first. A model with multi-token prediction layers needs no extra download. For EAGLE-3, check that the draft model supports your prompt length, and train a draft on your own traffic when acceptance is low. Google Cloud team Google Cloud 2025-11 and Modal with Decagon team Modal 2025-11 both trained their own drafts. Modal reported 38% longer acceptance than open-source drafts. Both are vendor reports.

Incidents

Speculation made a stress test slower. Issue #5274 (2025-04-11): EAGLE-3 on an A6000 lowered throughput at concurrency 20 from 217.8 to 164.4 tok/s. It raised batch-1 speed from 42.0 to 92.5 tok/s. The fix was to run speculation only at small batch.

The draft could not read long prompts. Issue #6783 (2025-05-30): Llama-3.3-70B FP8 with EAGLE-3 accepted 3.2 tokens at 1k input, 1.6 at 3k, and 1.2 at 10k. At 10k it produced 63.9 tok/s against 102.7 without speculation. The draft config had 2,048 positions and no rope scaling. Adding Llama 3 rope scaling and 131,072 positions to the draft config fixed it.

A loader bug hid the expert weights. David Noel Ng field David Noel Ng 2026-08 found SGLang slower than vLLM on DeepSeek-V4-Flash. Acceptance was 1.3 to 1.8 tokens, and only a log warning pointed at the cause. The draft loader never mapped the shared expert weights into the fused slot. After a local fix, decode went from 115.7 to 176.0 tok/s, and PR #33276 fixed it upstream.

Sources:python/sglang/srt/speculative/eagle_worker_v2.py (forward_batch_generation), speculative/eagle_utils.py (build_tree_kernel_efficient, eagle_sample), arg_groups/speculative_hook.py (_auto_choose_speculative_params, _handle_eagle_family, _handle_ngram, _resolve_speculative_algorithm_alias), arg_groups/fields/spec.py, managers/scheduler_components/metrics_reporter.py, and observability/metrics_collector.py at v0.5.22; issues #5274 and #6783; PR #33276; field reports by Ant Group on the LMSYS blog (2025-09-26) and David Noel Ng (2026-08-04); vendor reports by Google Cloud (2025-11-25) and Modal with Decagon (2025-11-13).

Your agent asks the model for a tool call and parses the reply with json.loads. Most replies parse. A few have a missing brace, a trailing comment, or a field the schema does not allow, and the agent retries. Each retry costs a full generation and a full round trip, and the slow tail of your latency is now made of retries.

When you finish this section, you can constrain output with a JSON schema, a regex, or an EBNF grammar. You can also explain what the mask costs, read the grammar queue, and decide what to check after generation.

The obvious fix, and why it fails

The obvious fix is to validate after generation and retry on failure. That works and stays slow. It also fails in the cases you care about most. A model that broke the schema once on a prompt tends to break it again on that prompt.

Constrained decoding: removing every token that would break the grammar before the sampler picks one. The output then satisfies every rule that the backend enforces, so those rules become a property of the server.

What SGLang does

A request carries one constraint in its sampling parameters: json_schema, regex, ebnf, or structural_tag. The OpenAI chat API maps response_format to the same fields. --grammar-backend picks the engine that compiles it. The default at v0.5.22 is xgrammar, and outlines, llguidance, and none are the other choices.

1. Look up. The grammar manager builds a key from the constraint type and its text, such as ("json", schema). A compiled grammar for that key comes back from the cache at once.
2. Compile. On a miss, compilation starts in a thread pool and returns a future. The request moves to the grammar queue instead of the waiting queue.
3. Admit. Each scheduler loop polls the futures. With tensor or data parallelism, a request is admitted only when its grammar is ready on every rank. It fails if compilation failed on any rank. An invalid grammar is cached as invalid, so the next request with the same schema fails at once.
4. Mask. At every step the grammar fills a vocabulary bitmask, and the model runner applies it to the logits before sampling. With the overlap scheduler, the mask for the next step is built while the GPU runs the current one.
5. Advance. After sampling, the grammar accepts the chosen token. With speculative decoding, it rolls back the rejected draft tokens.

On Apple silicon, the MLX runtime applies grammar masks only with --mlx-enable-sampling. Without that flag every decode step takes the highest logit, as section 4.6 shows from the help text.

Grammar queue: the list of requests whose grammar is still compiling. /metrics exports its length as sglang:num_grammar_queue_reqs.

Jump-forward is not active at v0.5.22

The SGLang paper describes jump-forward decoding. When the grammar allows only one continuation, such as the fixed text {"name": ", the server writes it without running the model. The grammar classes at v0.5.22 still have the jump-forward methods. A search of the runtime finds no caller outside the grammar classes. The scheduler does not use them, so expect one forward per token even inside fixed text. We read this in the code and did not run a timing test.

What to measure

Measure three things on your own schemas. First, the time to first token for the first request with a new schema against the second request with the same schema. The difference is compile time. Second, inter-token latency with and without the constraint at your real concurrency. The mask costs CPU time on every step. Third, the grammar queue length under load.

A field report by Varad More field Varad More 2026-04 measured structured generation on one A10G across 16 models from 2B to 9B. On 14 of the 16, vLLM was faster. The report does not separate compile time from per-token cost, so read it as a reason to measure, not as a verdict.

Our position

Constrain every response that a program parses. A mask is cheaper than a retry.

Do not trust the grammar backend to enforce every JSON Schema keyword. Test each keyword your consumer depends on, such as minLength, pattern, minimum, and enum, with a request that tries to break it. Keep a cheap validator behind the server for the fields that matter.

Send a few requests with each common schema after every deploy, so the first user request does not pay the compile time.

Incidents

A schema rule was dropped without an error. Issue #37707 (2026-09-03): with xgrammar, a string field accepted values shorter than its minLength when the same field also had a pattern. The same report says outlines dropped pattern and numeric bounds. The issue was open when this manual was verified.

Grammar work grew with output length under speculation. Issue #38863 (2026-09-10): with speculative decoding, the host CPU time to build bitmasks grew with the length of the output. The cause was a rollback method that copied a debug list on every call. The issue was open as a performance report.

Sources:capture/out/serve-help.txt, python/sglang/srt/constrained/grammar_manager.py (process_req_with_grammar, get_ready_grammar_requests), constrained/base_grammar_backend.py, constrained/xgrammar_backend.py, model_executor/model_runner.py (_preprocess_logits), sampling/sampling_params.py, arg_groups/serving_hook.py (handle_grammar_backend), arg_groups/choices.py (GRAMMAR_BACKEND_CHOICES), and observability/metrics_collector.py at v0.5.22; SGLang paper (arXiv 2312.07104); issues #37707 and #38863; field report by Varad More (2026-04-22).

You run an evaluation twice at temperature 0 and the scores differ. You send one prompt alone and then inside a busy batch, and the second answer diverges after forty tokens. Nothing in your request changed. The server changed the batch your request ran in, and the batch changed the arithmetic.

When you finish this section, you can explain why batching changes output and what deterministic mode switches. You can also read the logprobs SGLang returns and test whether your model is deterministic.

The cost: addition order depends on the batch

Floating-point addition is not associative. Summing the same numbers in a different order can change the last bits of the result. A matrix multiply, a softmax, or a norm splits its sums into tiles, and many kernels pick the tile split from the batch size. A request in a batch of 1 and the same request in a batch of 37 go through different splits.

A difference in the last bit can change which token has the highest logit. After one different token, every later token is conditioned on a different text. Thinking Machines described this effect in a 2025 post on nondeterminism in inference field Thinking Machines 2025-09, and SGLang built its deterministic mode on the same approach.

Batch invariance: a kernel property. The result for one row does not depend on the other rows in the batch, because the reduction order is fixed.

What the flag switches

--enable-deterministic-inference is off by default. --rl-on-policy-target fsdp turns it on too. When it is on, startup does the following:

1. It replaces the PyTorch operators aten::mm, addmm, _log_softmax, mean.dim, rms_norm, and bmm with batch-invariant kernels.
2. It forces the pytorch sampling backend.
3. With tensor parallelism on CUDA, it sets NCCL_ALGO=allreduce:tree, so the all-reduce sums in a fixed order. On AMD it uses a one-stage all-reduce.
4. It accepts only the attention backends ascend, fa3, fa4, flashinfer, intel_xpu, and triton.
5. It turns the radix cache off unless the attention backend is ascend, fa3, fa4, or triton.
6. It gives every request without a sampling_seed the seed 42.

The fifth point matters most for a server. With flashinfer, deterministic mode turns off prefix caching. Every repeated prefix pays its full prefill again. Section 3.3 explains what that cache saves.

Logprobs are temperature-scaled

Logprob: the log probability of a token under the distribution the server sampled from. SGLang returns it when a request sets return_logprob or the OpenAI logprobs field.

The sampler computes returned logprobs as log_softmax(logits / temperature) when the batch is not all greedy. It returns log_softmax(logits) without the temperature only for all-greedy batches or when the environment variable SGLANG_RETURN_ORIGINAL_LOGPROB is set. A request at temperature 0.7 gets logprobs of the sharpened distribution.

That is correct for sampling, and it surprises anyone who compares those numbers with a model run at temperature 1. A reinforcement learning trainer that computes logprobs from the raw logits sees a mismatch on every token, and the mismatch looks like a numerical bug. Section 7.2 shows why that matters for training.

On Apple silicon

Our capture ran on the MLX runtime, and MLX treats sampling differently. The help text for --mlx-enable-sampling says what changes.

  --mlx-enable-sampling
                        MLX backend only: sample decode tokens (temperature /
                        top-k / top-p / min-p) instead of greedy argmax.
…
                        exact argmax behavior. Also enables on the MLX path:
                        grammar vocab masks and custom logit processors (these
                        break decode chaining per step; custom processors run
                        on pure-decode steps only), logit_bias, output
                        logprobs (sampled token / top-k / token_ids; prompt
                        input logprobs are not computed), NaN sanitization
                        (SGLANG_SANITIZE_NAN_LOGITS), and per-request
                        sampling_seed under --enable-deterministic-inference
                        (deterministic within MLX only). Penalties are not
                        applied.

Without that flag, every decode step on MLX takes the token with the highest logit, whatever temperature the request asks for. Grammar masks and output logprobs also need the flag, and penalties never apply on MLX. Our capture kit sends temperature 0, so its results do not depend on this flag.

What to measure

SGLang includes a test that sends one prompt in random batch sizes and counts distinct outputs. It has three modes: single, prefix for shared prefixes, and radix_cache for cached against uncached prefill.

python3 -m sglang.test.test_deterministic --n-trials 50 --test-mode single
python3 -m sglang.test.test_deterministic --n-start 1 --n-trials 50 --test-mode prefix

Run it against your model, your attention backend, and your parallel layout. One distinct output per prompt is the pass condition. Then measure throughput with the flag on and off, because batch-invariant kernels give up some speed.

Our position

Treat --enable-deterministic-inference as a promise for one model on one backend, and test that promise. Turn it on for reinforcement learning rollouts and for evaluations you want to reproduce. Leave it off for production traffic, where prefix caching and faster kernels are worth more than identical bits.

Set sampling_seed on each request that must reproduce. Seed 42 for everyone makes every sampled request draw the same random numbers.

Incidents

The flag was accepted and the output still varied. Issue #43055 (2026-10-08): gpt-oss-20b gave several different greedy outputs for one prompt in prefix mode. Comments traced it to the deterministic Triton extend kernel on sliding-window layers. It aligned KV tiles to the trimmed prefix and not to absolute positions. The issue was open when this manual was verified.

Some architectures are not covered yet. Issue #43413 reported that a Mamba2 hybrid model was not batch-invariant. Issues #36174 and #31261 list DeepSeek-V4 and Qwen3.5 as not yet supported in deterministic mode.

A restart changed greedy output. Issue #39597 (2026-09-15): with the DFLASH speculative algorithm, the same prompt gave different text after a restart. FlashInfer autotuning picked different kernel tactics on each start. The workaround was --disable-flashinfer-autotune or deterministic mode.

Sources:python/sglang/srt/batch_invariant_ops/batch_invariant_ops.py (enable_batch_invariant_mode), arg_groups/attention_hook.py (handle_deterministic_inference), arg_groups/choices.py (DETERMINISTIC_ATTENTION_BACKEND_CHOICES, RADIX_SUPPORTED_DETERMINISTIC_ATTENTION_BACKEND), sampling/sampling_batch_info.py, layers/sampler.py, and python/sglang/test/test_deterministic.py at v0.5.22; capture/out/serve-help.txt; issues #43055, #43413, #36174, #31261, and #39597; Thinking Machines, Defeating Nondeterminism in LLM Inference (2025-09).

A teammate posts "SGLang does 255 tokens per second on my laptop" in the team channel. The number is true. It came from our capture. It is also useless on its own. The same server gave that throughput with a median first token of 0.8 s at one load and 3.9 s at another. A throughput number without the latency that came with it describes nothing a user feels.

When you finish this section, you can run sglang.benchmark.serving with settings you can defend and read every line of its report. You can find the knee where more concurrency stops paying. You can also say which of your own numbers will transfer to a GPU and which will not.

The command we ran

The server is one process with the model mlx-community/Qwen3-0.6B-4bit on the capture Mac. Every run in this section talks to this one server.

SGLANG_USE_MLX=1 sglang serve --model-path mlx-community/Qwen3-0.6B-4bit --disable-cuda-graph --host 127.0.0.1 --port 30000 --enable-cache-report --enable-metrics --mem-fraction-static 0.4

Two flags in that command need a note. SGLang's Apple silicon guide still passes --disable-cuda-graph, and the v0.5.22 help marks it deprecated in favor of --cuda-graph-backend-{decode,prefill}=disabled. On the MLX backend, decode is greedy unless you add --mlx-enable-sampling, so our runs measured the greedy path. The benchmark sends temperature 0 by default, so the client and the server agree here.

The client is the benchmark module that comes with SGLang. We ran it three times, with a concurrency cap of 1, 8, and 32. The run at 8 looks like this:

python -m sglang.benchmark.serving --backend sglang --port 30000 \
  --dataset-name random --random-input-len 512 --random-output-len 128 \
  --random-range-ratio 1.0 --num-prompts 32 --max-concurrency 8 --seed 7

The path changed. The benchmark tools moved into the sglang.benchmark package in 2026-06. python -m sglang.bench_serving still runs as a deprecated alias that imports the new module. python -m sglang.bench_one_batch and python -m sglang.bench_offline_throughput do not run at v0.5.22. PR #40795 deleted both aliases on 2026-09-22 code. If a tutorial uses those two paths, it was written for an older SGLang, and its other flags are suspect too.

Four defaults that change the result

The client makes choices before it measures anything. Each one changes the number you report, and most published benchmarks do not say which ones they kept.

The first row is the trap. With the default ratio, --random-input-len 512 gives an average prompt near 256 tokens. Your report then says 512 and describes half the work. We set the ratio to 1.0, and the report confirms it. 32 prompts of 512 tokens give Total input tokens: 16384. With ignore_eos, the run gives Total generated tokens: 4096, exactly 32 times 128.

Our position: pass --random-range-ratio 1.0 every time, or write the length distribution next to the result. A benchmark that hides its lengths cannot be compared with anything.

Read the report, block by block

============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 8
Successful requests:                     32
Benchmark duration (s):                  16.02
Total input tokens:                      16384
Total generated tokens:                  4096
Request throughput (req/s):              2.00
Output token throughput (tok/s):         255.63
Peak output token throughput (tok/s):    440.00
Peak concurrent requests:                16
Total token throughput (tok/s):          1278.13
Concurrency:                             7.96
…
Median TTFT (ms):                        785.79
P99 TTFT (ms):                           2180.87
…
Mean TPOT (ms):                          23.02
Median TPOT (ms):                        22.52
…
Median ITL (ms):                         20.35
P99 ITL (ms):                            78.79
Max ITL (ms):                            1948.37

The function calculate_metrics in the benchmark module computes every line. Five of them are easy to misread.

Concurrency: the sum of every request's end-to-end latency divided by the run duration. It is the average number of requests in progress, measured after the fact. Here it is 7.96 against a cap of 8. It tells you the cap was reached, and nothing more.

Peak concurrent requests: the run is cut into one-second buckets, and each request counts in every bucket that its interval touches. The line reports the largest bucket count. Two requests that run one after the other inside the same second both count, so the number can exceed the cap. That is why the report says 16 at a cap of 8, and 64 at a cap of 32.

TTFT: the time from sending the request to receiving the first streamed chunk. It contains queueing, prefill, and the network.

TPOT: for each request, the time after the first token divided by the remaining tokens, (latency - ttft) / (output_len - 1). The report gives the mean and percentiles over requests. ITL: every gap between two streamed chunks, pooled across all requests. A chunk with several tokens counts as one gap.

Max ITL: the single longest gap, here 1,948 ms. One request waited almost two seconds between two tokens while new prompts ran their prefill. The median ITL of 20 ms hides that stall. The P99 of 79 ms shows part of it.

The sweep and the knee

One run tells you one point. Three runs at caps of 1, 8, and 32 tell you the shape.

We measured this on the capture Mac. From 1 to 8, output throughput rises 46 percent and the median first token gets 4.8 times slower. From 8 to 32, throughput falls about 1 percent. The median first token gets 5 times slower, and the median time per token gets 4.2 times slower.

The knee is the cap after which throughput stops rising. On this machine it is at or below 8. Past the knee, the server has no idle compute left to give. New requests only wait for a share of the same decode steps. The scheduler in section 2.2 accepts them anyway, because its admission limit on this Mac was max_running_requests=4096.

What transfers to a GPU. The absolute numbers do not. A 0.6B model on Apple silicon says nothing about an H100. The shape does transfer. Every server has a knee, the latency past it grows faster than the throughput, and the default admission limit sits far above it.

Our position

Report the highest concurrency that still meets your latency target, and report its throughput together with p50 and p99 TTFT and TPOT. "255 tok/s at a cap of 8, with a median first token of 0.79 s" is a result. "255 tok/s" is a slogan.

Then enforce the knee outside the engine. The engine's own limit is set from memory, and memory runs out long after latency is unacceptable. Put the cap in your client, your gateway, or --max-running-requests, and size it from the sweep.

The incident: a hand-written script

Issue #3196 reported 13 tok/s for DeepSeek V3 on eight H200 GPUs, against 60 tok/s from the DeepSeek API issue #3196. Another user reached 33 tok/s on two H100 nodes. The measuring script counted words split on spaces and included the first-token time in the rate. A word is not a token, and TTFT is not decode time, so the script reported less than the server did. The fix was to measure with the benchmark client, which counts tokens and keeps TTFT and TPOT apart.

Sources:python/sglang/benchmark/serving.py (calculate_metrics, argument defaults), python/sglang/benchmark/datasets/common.py (compute_random_lens), python/sglang/bench_serving.py, PR #40795, all at v0.5.22 (baa9befa); capture/out/serve-help.txt, capture/out/server-default.command.txt, capture/out/server-default.log, capture/out/bench-random-c1.txt, capture/out/bench-random-c8.txt, capture/out/bench-random-c32.txt; issue #3196.

Your dashboard says the prefix cache hit rate is 94 percent. Over the same run, the server computed 104,057 prompt tokens and took 182,079 from the cache, a real hit rate of 64 percent. Both numbers came from one scrape of the server in our capture. One is a gauge that remembers the last prefill batch it logged. The other is a pair of counters that remember everything.

When you finish this section, you can turn the endpoint on and tell a gauge from a counter and a histogram. You can compute a hit rate that means something, and write six alerts for a server past its knee.

Turn it on first

The endpoint is off by default. The field enable_metrics defaults to False at v0.5.22 code, and without --enable-metrics the path /metrics returns nothing useful. Our capture server started with the flag, as section 5.1 shows. After the default experiments, it exported 81 metric families in 603 sample lines that start with sglang:.

Our position: start every server with --enable-metrics, including the one on your laptop. The cost is a few counters per request. The cost of an incident without history is the whole investigation.

Three kinds of series

Gauge: a value the scheduler overwrites on each log interval, such as the number of running requests. A scrape sees the latest value only. Counter: a total that only grows, such as the number of prompt tokens processed. You read it with a rate over a window. Histogram: a set of counters, one per bucket, that counts how many requests fell under each limit. You read percentiles from it with a rate over a window.

The snapshot below was taken when the server was idle, after all default experiments.

sglang:num_running_reqs{…} 0.0
sglang:num_queue_reqs{…} 0.0
sglang:gen_throughput{…} 0.0
sglang:token_usage{…} 0.0
sglang:cache_hit_rate{…} 0.9418739997090062
sglang:num_retracted_reqs{…} 0.0

Five gauges show an idle server. One does not. The cache hit rate still holds the value from the last prefill batch that ran, which was the shared-prefix benchmark. A gauge answers "what happened in the last interval", and when nothing happens, some gauges decay and some keep their last value.

The cache hit rate, two ways

The scheduler computes the gauge from the prefill tokens of one log interval. The value is hit tokens divided by hit tokens plus computed tokens. The counters below add up the same two quantities since the server started.

# HELP sglang:prefill_effective_tokens_total Effective prefill tokens with retracted-request re-counts excluded, updated on each log interval. mode: device_hit, host_hit, storage_hit, input. Windowed prefix cache hit rate = rate(sum of *_hit) / rate(sum of all modes); per-tier rate uses a single *_hit mode in the numerator.
sglang:prefill_effective_tokens_total{…mode="input"…} 104057.0
sglang:prefill_effective_tokens_total{…mode="device_hit"…} 182079.0
sglang:prefill_effective_tokens_total{…mode="host_hit"…} 0.0
sglang:prefill_effective_tokens_total{…mode="storage_hit"…} 0.0

The HELP text says that re-counted tokens are excluded. In the code, that is true only for the input mode. The three hit modes add the raw hit counts, so after a retraction they can count the same tokens twice code. Our run had no retractions, so the numbers below are exact.

From the counters, the hit rate over the whole run is 182,079 divided by 286,136, which is 0.636. The gauge says 0.942. Both are correct, and they answer different questions. The gauge describes the shared-prefix benchmark at the end of the run. The counters describe all of it, including the random-prompt benchmarks that could hit nothing.

The HELP text of the counter gives the formula for a windowed rate. In PromQL it reads like this:

sum(rate(sglang:prefill_effective_tokens_total{mode=~".*_hit"}[5m]))
  / sum(rate(sglang:prefill_effective_tokens_total[5m]))

The server with --disable-radix-cache reported a gauge of 0.0 in its own snapshot, so the gauge does reach zero when nothing can hit. A low value from the counters is the signal section 3.4 teaches you to act on.

What to alert on

These six series cover the failures in this manual. Every name below appears in our capture.

The p99 of a histogram comes from its buckets:

histogram_quantile(0.99,
  sum by (le) (rate(sglang:time_to_first_token_seconds_bucket[5m])))

We measured both histograms in the same snapshot. The queue time histogram has a sum of 251.7 s over 266 requests, a mean of 0.95 s. The streaming TTFT histogram has a sum of 612.7 s over 261 requests, a mean of 2.35 s. The two series count different requests, so their ratio is an estimate, not a measured share. It suggests that queueing was a large part of the first-token time on this Mac. For an exact share, compare queue time and TTFT of the same requests.

sglang:queue_time_seconds_sum{…} 251.74525608913973
sglang:queue_time_seconds_count{…} 266.0
sglang:time_to_first_token_seconds_sum{…is_streaming="true"…} 612.7282021734864
sglang:time_to_first_token_seconds_count{…is_streaming="true"…} 261.0

Our position

Alert on latency and activity with histogram percentiles and counter rates over a window. Alert on state with a gauge only when the condition lasts, such as num_queue_reqs above 0 or token_usage near 1.0 for several minutes. A single gauge scrape describes the last few seconds. When a gauge and a counter disagree, as the cache hit rate did here, the counter is the one to trust.

Read every new metric in its HELP text and in the code before you build on it. Not every gauge works.

The incidents

A gauge that never moves. The scheduler code at v0.5.22 carries a comment that sglang:utilization stays at 0 since a regression it links to #22713 code. Our capture agrees: the gauge reads 0.0 after a run that kept the server busy for minutes. A dashboard panel built on it would show an idle fleet during an outage.

A health probe that piled up work. Issue #35884 reported that a timed-out /health call did not cancel its request in the scheduler issue #35884. Each failed probe left one more orphaned request behind, and under load the probes added to the load they were trying to detect. A related report, issue #22511, shows the message "health check failed" under high load on a large model issue #22511. Watch the queue and latency series, and treat a failing probe as one more symptom.

Sources:python/sglang/srt/arg_groups/fields/observability.py (enable_metrics), python/sglang/srt/managers/scheduler_components/metrics_reporter.py (cache hit rate and utilization), python/sglang/srt/observability/metrics_collector.py, all at v0.5.22 (baa9befa); capture/out/metrics.txt, capture/out/no-radix-cache-metrics.txt, capture/out/server-default.command.txt; issues #35884 and #22511.

The most repeated SGLang number on the web is "29 percent faster than vLLM": 16,215 tokens per second against 12,553. It comes from one offline test by the analyst firm AIMultiple, updated 2026-01-25 field AIMultiple 2026-01. That page lists SGLang as "v0.2.3" next to vLLM 0.11.0, a gap of about two years of releases. At least five later posts repeat the pair, some under the names of other companies. None of them reran the test.

When you finish this section, you can take a published SGLang benchmark apart into its five inputs. You can decide whether it applies to you and name the mechanism behind it. You can also say where the independent tests agree, which is a shorter list than the headlines suggest.

The five inputs behind every result

A benchmark result is a function of five inputs. Most disagreements between published reports trace back to one of them.

1. Workload shape. Input length, output length, concurrency, and the share of the prompt that repeats. The radix cache helps only when prefixes repeat. Long inputs stress prefill. High concurrency stresses the scheduler and the KV pool.
2. Memory budget. SGLang computes --mem-fraction-static from the GPU when you leave it unset. vLLM uses --gpu-memory-utilization 0.9. A test that sets one and not the other compares two different KV pools.
3. CUDA graph ceiling. The decode step replays a captured CUDA graph only up to a batch size limit. Above it, decode runs eager, one kernel launch at a time. Section 4.1 explains the mechanism.
4. Kernel path. The attention backend, the MoE kernel, and the quantization format. Several reports below were decided by a kernel that did or did not exist for one GPU.
5. Version and date. One analyst measured the same model on the same AMD GPU at 20.4 and then 2,256 tokens per second per GPU, 26 days apart.

The code sets the third input from GPU memory at startup. Take an 80 GB H100 with tensor parallel size below 4. The default decode graph limit is 256, and the chunked prefill size is 8,192. On a 32 GB card the limit is 48, raised from 24 on 2026-09-04. The code comment on that change says the old cap "collapsed high-concurrency throughput vs vLLM" code.

Six questions before you trust a number

1. Is the SGLang version stated? Without it, the result describes an unknown build.
2. Does the workload match yours? Unique prompts say nothing about a chat server with a long shared system prompt.
3. Were the memory budgets matched across engines?
4. Did the concurrency pass the decode graph limit? If yes, the test measured eager decode.
5. Are the flags mapped correctly? The same idea has different flag names in each engine.
6. Who paid for the test? A GPU cloud, a hardware vendor, a competitor, or the SGLang team each has a reason to like one answer.

Where independent tests agree

These reports come from people outside the SGLang team. Each row names the mechanism from earlier sections that explains it.

The links: Spheron field Spheron 2026, Modal field Modal 2025-06, Varad More field Varad-More 2026-04, Steve Rawlinson field Rawlinson 2025-12, RunPod field RunPod 2025-06, Jarvislabs field Jarvislabs 2026-05, SemiAnalysis field SemiAnalysis 2025-10, Crusoe field Crusoe 2026-09, and InferenceX field InferenceX 2026-05.

Read the caveats with the numbers. Rawlinson left SGLang memory at its default and vLLM at 0.9, so the two KV pools differed. Jarvislabs did not record either engine version. Spheron, RunPod, Modal, and Jarvislabs sell GPU time, which is a reason to run tests and no reason to favor one engine. The Crusoe result is the only one here that passed an external audit. Its p99 time per token was 79.98 ms against a limit of 80.

Fragile on new hardware

The same reports show one more pattern. A Blackwell workstation card ran SGLang only with --disable-cuda-graph, according to joshua8.ai field joshua8 2026-01. A mixed cluster crashed when an older card joined, because the FP4 path required compute capability 10.0. Pivot News summarized that community post field Pivot-News 2026-05. The fast paths are written per GPU generation, and the paths for other GPUs are fewer. On new hardware, expect the first SGLang release to be slow or broken, and measure again every week.

Our position

For a dense model with unique prompts, the choice of engine is not a performance decision. Decide on model support, startup time, and the hardware you have. For repeated prefixes, SGLang has a real advantage that is smaller than the stories about it. For DeepSeek-class MoE models on many GPUs, start from SGLang, because that is where the field runs them.

Above all, do not import a result. Run the sweep from section 5.1 on your own workload. A published number tells you which mechanism to test, and nothing more.

The incidents

A memory result that was a flag error. In vLLM discussion #17221, a user first saw SGLang use 7 GB against vLLM's 21 GB on one A10 field vLLM-discussion 2025-04. Then the user found that the flag that matches vLLM's --max-model-len is --context-length, not --max-total-tokens. With the right flag, memory use was about the same.

Advice that no longer parses. Field reports from 2026 advise raising the decode graph limit with --cuda-graph-max-bs. At v0.5.22 that flag is split into --cuda-graph-max-bs-decode and --cuda-graph-max-bs-prefill. The code rejects the old spelling. The argument parser of sglang serve stops with ambiguous option: --cuda-graph-max-bs could match --cuda-graph-max-bs-decode, --cuda-graph-max-bs-prefill. A tuning recipe is tied to its version as tightly as a benchmark is.

The recycled number. The 29 percent pair from the start of this section was criticized by DeepInfra on 2026-08-04 for its version gap field DeepInfra 2026-08. The author answered on 2026-10-14 that the result "should not be extrapolated" to long context field AIMultiple 2026-10. The SGLang team made a larger claim in 2024: "up to 3.1x higher throughput on Llama-70B" team 2024-07. No independent report found in this research reproduced a gap of that size on a dense model with unique prompts.

Sources:python/sglang/srt/arg_groups/memory_hook.py (GPU memory tiers), python/sglang/srt/arg_groups/fields/exec_.py (cuda_graph_max_bs_decode), python/sglang/srt/arg_groups/fields/schedule.py, all at v0.5.22 (baa9befa); the field reports linked in the text, fetched 2026-10-16; aimultiple.com/inference-engines; deepinfra.com/blog/vllm-vs-sglang; lmsys.org/blog/2024-07-25-sglang-llama3; inferencex.semianalysis.com/blog/mi355x-deepseek-v4-pro-sglang-110x-in-26-days.

You have eight GPUs and one large model. The first command most people type is --tp-size 8, and the model loads. Then a teammate asks why the server holds so few tokens of KV cache for a DeepSeek model on 640 GB of memory. The answer is in how SGLang splits one model across GPUs, and in a flag whose meaning changed in v0.5.22.

When you finish this section, you can choose --tp-size, --dp-size, and --attn-dp-size for a model and a GPU count. You can also say which GPUs share which work, and read an old recipe that uses --enable-dp-attention.

One world, many ways to split it

Model world: the set of GPUs that together hold one copy of the model. SGLang builds it from two numbers only. The code raises an error when the world size is not tp_size times pp_size code.

Tensor parallelism (TP): each layer's weight matrices are cut into --tp-size slices, one per GPU. Every layer then ends with a collective operation that joins the slices. Pipeline parallelism (PP): whole layers go to different GPUs in order, and --pp-size sets the number of stages.

Everything else splits that same world. Attention data parallelism, expert parallelism, and context parallelism do not add GPUs. They regroup the GPUs of the TP group differently for different layers.

Keep one mental model, and every later flag reads clearly. TP size is the number of GPUs in one model copy. The other widths are ways to cut that set.

The default of every one of these flags is 1 code. So a plain sglang serve uses one GPU, one scheduler, and one copy of the model.

Two meanings of "data parallel"

Data parallelism (DP): several complete copies of the model, each with its own requests and its own KV cache. --dp-size 2 with --tp-size 4 uses eight GPUs as two worlds of four. A data parallel controller process gives each new request to one of the copies code. Its default method, auto, resolves to round_robin outside prefill-decode disaggregation.

Attention data parallelism: inside one TP group, the attention layers run data parallel while the feed-forward or expert layers stay split across all GPUs. Each attention group has its own scheduler, its own requests, and its own KV cache. --attn-dp-size 4 with --tp-size 8 gives four attention groups of two GPUs each.

Before v0.5.22, one flag carried both meanings. --dp-size counted model copies, unless --enable-dp-attention was also set, when it counted attention groups. Pull request #41818 gave the reason for the change in one sentence: "--dp-size means two different things... Every reader has to look at both flags to know which meaning it has." It was merged on 2026-09-30.

The code at v0.5.22 does three things with the old spelling code:

The warning names the new flag, and a comment in the code says to remove the old flag after 2026-12-31 code. Most DeepSeek recipes on the web still use the old spelling. They work today and stop working in a later release.

Why attention gets its own data parallel mode

DeepSeek models use multi-head latent attention (MLA). MLA stores one compressed latent vector per token and layer, which is one KV "head". Section 3.1 works through its size. Tensor parallelism cannot split one head across GPUs, so with --tp-size 8 every GPU stores the full latent cache for every token. Eight GPUs then hold eight copies of the same KV cache.

The SGLang team gave this reason when they added the mode in v0.4 team 2024-12. With attention data parallelism, each attention group keeps only its own requests. The KV capacity of the server grows with the number of groups. The team reported up to 1.9 times more decode throughput for DeepSeek models, which is their claim.

For a dense model with grouped-query attention (GQA), the reason does not hold. GQA has several KV heads, so tensor parallelism already splits the cache without copies.

What attention data parallelism changes for you

Turning on attention data parallelism changes two other settings without a flag of their own code:

• chunked_prefill_size is divided by --attn-dp-size. A chunk size that you tuned on one GPU becomes smaller per attention group. The server log prints the old and the new value.
• schedule_conservativeness is multiplied by 0.3. The scheduler then admits new requests with a smaller reservation for future tokens. Section 2.3 explains that reservation.

The first change is the one that surprises people. A team that measured TTFT with --chunked-prefill-size 8192 and then added --attn-dp-size 8 now runs chunks of 1024 tokens. Long prompts take more scheduler steps to prefill.

Our position

Use tensor parallelism to make the model fit, and nothing more. Add capacity with data parallel copies, inside one server with --dp-size or as separate servers behind the gateway in section 6.4. Use --attn-dp-size equal to --tp-size for MLA mixture-of-experts models such as DeepSeek and Kimi, and leave it at 1 for dense GQA models.

A field report supports the order. A student team tuned DeepSeek-R1 for a competition on H100 and H200 nodes field Lowndes 2025-11. Plain --tp 16 gave about 5.9k tokens per second offline. Their best layout, TP4 with DP4, PP2, and attention data parallelism on one H200 node, gave 17,417 tokens per second. Use their numbers as one data point, because the write-up explains --mem-fraction-static incorrectly.

Ant Group made the same kind of choice in production. Their DeepSeek-R1 decode runs on two nodes with 16 attention groups and 16-way expert parallelism field Ant-Group 2025-09.

Several nodes

One model world can span machines. Three flags describe it code:

Only node 0 runs the tokenizer and the HTTP API. Other nodes run their scheduler processes, a small health server, and nothing else code. Point your load balancer at node 0. Also keep adjacent ranks on the same machine, because TP groups are contiguous rank ranges and their collectives need the fast links inside one box.

A real incident shows how a network detail can stop all of this. A user ran attention data parallelism across nodes with an IPv6 address in --dist-init-addr (issue #19953). The server hung on its first message to the schedulers, and no request was ever served. The cause was a ZeroMQ socket that the controller bound without IPv6 turned on. A fix was proposed in the thread. We could not confirm that it was merged.

What to measure

Before you add GPUs, measure one copy with the benchmark in section 5.1. Watch sglang:num_queue_reqs and sglang:token_usage from section 5.2. A queue that grows while token usage stays low points to compute limits, and more copies help. Token usage near 1.0 points to KV memory limits, and a smaller KV cache per token helps first.

For attention data parallelism, watch sglang:dp_attention_token_imbalance_ratio. It reports the busiest group's tokens over the average, and 1.0 means balanced. Our /metrics capture lists it even on a single Mac, so your dashboards can include it before you need it.

Sources:python/sglang/srt/distributed/parallel_state.py (initialize_model_parallel), python/sglang/srt/arg_groups/fields/parallel.py, python/sglang/srt/arg_groups/parallel_hook.py (handle_deprecated_dp_attention, handle_data_parallelism), python/sglang/srt/server_args.py, python/sglang/srt/entrypoints/engine.py, and python/sglang/srt/managers/data_parallel_controller.py, all at v0.5.22. capture/out/metrics.txt. Pull request #41818, issue #19953. LMSYS, "SGLang v0.4", 2024-12-04. Luca Lowndes, "SGLang optimisations", 2025-11-07. Ant Group on the LMSYS blog, 2025-09-26.

A mixture-of-experts (MoE) model such as DeepSeek-V3 activates a few experts per token out of many. On paper that is cheap compute. In a server it is a traffic problem. Each token must travel to the GPUs that hold its chosen experts, and each result must travel back, in every MoE layer of every step. One overloaded GPU then makes all the others wait.

When you finish this section, you can follow one token through an MoE layer in SGLang. You can also choose an all-to-all backend and an expert parallel size, and decide when to turn on load balancing and batch overlap.

One MoE layer, five stages

Expert parallelism (EP): the experts of each MoE layer are divided among GPUs, and tokens move to the experts. --ep-size sets how many GPUs share the experts. Its default is 1 code.

SGLang cuts the MoE forward pass into stages with one owner each docs. The router picks experts for each token. The dispatcher moves tokens to the GPUs that hold those experts. A runner computes all local experts with grouped matrix multiplications. The dispatcher then moves results back and combines them.

Two flags pick the implementations code:

With none, SGLang moves tokens with all-reduce or all-gather collectives. That works for any layout and costs more bandwidth at scale. The all-to-all backends send each token only where it must go.

The layout rules that SGLang enforces

The all-to-all backends span the whole TP group. When you choose one of them, SGLang sets --ep-size equal to --tp-size and logs the change at info level code. You cannot run DeepEP across four of eight GPUs. A mixed layout with expert parallelism smaller than tensor parallelism works only with none docs.

DeepEP has two dispatch modes. Normal mode sends large prefill batches with high throughput, and it cannot be captured in a CUDA graph. Low-latency mode sends small decode batches and works inside CUDA graphs. The default auto uses each mode where it fits. If you force --deepep-mode normal, SGLang turns off decode CUDA graphs and prints one warning line code. Section 4.1 shows what decode without graphs costs.

Choose the expert parallel size from your batch

Wider expert parallelism spreads experts thinner and moves more traffic over slower links. Ant Group measured this on H20 GPUs with DeepSeek-R1 decode field Ant-Group 2025-09:

EP32 won at the small batch and lost by 13 percent at the batch they run in production. They explained the result with traffic: about half of the MoE traffic stays on NVLink at EP16, against about a quarter at EP32. They also chose EP16 to limit how many requests one failed GPU takes down. Their post appeared on the LMSYS blog, and the measurements are theirs.

Expert load balancing

Real traffic does not spread evenly over experts. Some experts get most of the tokens, and the GPUs that hold them become the slowest rank in every step.

EPLB: the expert parallelism load balancer. It records which experts each forward pass uses. Every --eplb-rebalance-num-iterations passes, 1000 by default, it computes a new placement and moves expert weights code. --ep-num-redundant-experts adds extra copies of popular experts, and its default is 0. A threshold flag skips the rebalance when GPU utilization is already high. Its default of 1.0 means that it never skips.

EPLB is off by default (--enable-eplb is false). Turn it on when your measurements show imbalance, not before. Run with --expert-balancedness-report-mode prometheus and watch sglang:eplb_balancedness code. The SGLang team reported EPLB speedups of 1.49 times for prefill and 2.54 times for decode team 2025-05. Those numbers came from a simulated imbalance, so expect less on traffic that is already balanced.

Overlapping communication with compute

Two-batch overlap (TBO): the scheduler cuts a batch into two micro-batches. One micro-batch runs attention while the other one waits on dispatch or combine. Turn it on with --enable-two-batch-overlap. It is off by default code.

For prefill, SGLang splits by token count. If no split puts at least 48 percent of the tokens on each side, it cuts one sequence across both halves. That is --tbo-token-distribution-threshold, 0.48 by default.

The SGLang docs say that TBO can "unlock up to 2x throughput" docs. The blog post that the docs link to measured 27 to 35 percent on 96 H100 GPUs team 2025-05. Plan for the measured number.

Ant Group turned TBO down on H20 field Ant-Group 2025-09. It helped only at batch sizes of 64 or 128, and at those sizes their time per output token missed its target. They used single-batch overlap instead, --enable-single-batch-overlap, which overlaps shared experts with the combine step. They measured 8 to 10 percent at batches of 20 to 56.

Our position

Leave --deepep-mode at auto in a server that does both prefill and decode. Choose --ep-size from the batch size that meets your latency target, not from the largest published run. Turn on EPLB only after sglang:eplb_balancedness shows imbalance. Treat TBO as a throughput feature for large decode batches, and measure it against your time per output token before you keep it.

Two field reports at full scale

One result in this part passed an external audit. Crusoe submitted DeepSeek-R1 on 512 AMD MI355X GPUs to MLPerf Inference v6.1 with SGLang 0.5.15.post1 field Crusoe 2026-09. Each node ran TP8 with EP8 and eight attention groups. The offline score was about 2.90 million tokens per second. In the server scenario, p99 time per output token was 79.98 ms against a limit of 80 ms. Their write-up lists the flags.

The SGLang team's 96-GPU run reported 52.3k input and 22.3k output tokens per second per node team 2025-05. It used 12 nodes of eight H100 GPUs, with prefill and decode on separate nodes. Section 6.3 explains that split.

A real incident: one GPU runs out of memory

A team ran DeepSeek V3 with four prefill and six decode servers and large expert parallelism (issue #9787). Under load, one prefill rank's GPU utilization dropped to zero, and then every prefill process crashed. The thread traced it to DeepEP dispatch. Imbalanced routing sent too many tokens to one GPU, and that GPU ran out of memory. The suggested fixes were a smaller chunked prefill per attention group, a tuned initial expert placement, and EPLB. The lesson is general: in MoE serving, the busiest GPU sets both your speed and your memory limit.

Sources:python/sglang/srt/arg_groups/fields/exec_.py, python/sglang/srt/arg_groups/overrides.py (_a2a_ep_size), python/sglang/srt/arg_groups/moe_hook.py, python/sglang/srt/eplb/eplb_manager.py, python/sglang/srt/batch_overlap/two_batch_overlap.py, python/sglang/srt/observability/metrics_collector.py, and docs/docs/advanced_features/expert_parallelism.mdx, all at v0.5.22. LMSYS, "Deploying DeepSeek with PD Disaggregation and Large-Scale Expert Parallelism on 96 H100 GPUs", 2025-05-05. Ant Group on the LMSYS blog, 2025-09-26. Crusoe, MLPerf Inference v6.1 results, 2026-09-16. Issue #9787.

Your chat users see smooth streaming until someone pastes a long document. Then every stream pauses while the server prefills the document. Prefill is compute work on many tokens at once, and decode is one token per request per step. On one server, they share the same GPU steps, and the long prefill takes the step. Chunked prefill makes the pause shorter but cannot remove it.

When you finish this section, you can explain how SGLang moves a request's KV cache from a prefill server to a decode server. You can also start the smallest disaggregated setup and decide from measurements whether you need it.

Two servers, one request

Prefill-decode disaggregation (PD): two kinds of SGLang servers, started with --disaggregation-mode prefill and --disaggregation-mode decode. The default is null, which means one server does both code. A prefill server computes the prompt's KV cache and the first token. A decode server receives that KV cache and generates the rest.

The SGLang docs give two reasons for the split docs. Prefill batches interrupt decode batches. With attention data parallelism, one group can prefill while another decodes, so the decoding group waits for the slowest group in each step. Separate servers remove both kinds of waiting. Each pool can also use its own parallel layout, for example a wide TP for prefill and attention data parallelism for decode.

How the KV cache moves

The gateway in section 6.4 starts every exchange. It picks one prefill server and one decode server. It adds three fields to the request: the prefill server's bootstrap host, its bootstrap port, and a random 63-bit number called the bootstrap room. Then it sends the same request to both servers at the same time code.

Bootstrap room: the number that both servers use to find each other's half of one request. Think of it as a reservation number: the decode server books KV slots, and the prefill server writes into them.

1. The decode server creates a receiver, finds the prefill rank through the bootstrap server, and reserves KV slots for the prompt.
2. It sends the addresses of those slots to the prefill side.
3. The prefill server runs a normal prefill forward pass for the prompt.
4. It writes the KV pages into the decode server's reserved slots. With the default Mooncake backend, this is a one-sided RDMA write. Mooncake groups contiguous pages on both sides into one batch transfer code.
5. The decode server sees the transfer finish and adds the request to its running batch without another prefill.
6. The client stream comes from the decode server.

The bootstrap server runs on the prefill host. Its default port is 8998 code. Each side tracks a transfer with five states: Failed, Bootstrapping, WaitingForInput, Transferring, and Success code.

Settings that matter

All defaults come from the v0.5.22 code code. mooncake_tcp exists for clusters without RDMA network cards. We did not measure its speed.

Do not send requests straight to a prefill or decode server. The scheduler asserts that a bootstrap room exists, and its error message says "Do not send requests directly to prefill or decode instances; send to the router instead" code.

Our position

Do not start with disaggregation. Start with one server, then add data parallel copies behind the gateway. Split prefill from decode only after you measured that prefill interference hurts your inter-token latency at the concurrency you serve. Disaggregation adds a second set of failures: the bootstrap step, the RDMA transfer, two timeouts, and a router that must pair servers correctly.

A real incident shows the cost of skipping that measurement. A user ran one prefill and one decode server, each with TP4 (issue #12591). Time to first token was higher than on one TP4 server without disaggregation. The cause was an old Mooncake transfer engine. Upgrading it fixed the problem, and a maintainer also suggested SGLANG_DISAGGREGATION_QUEUE_SIZE=1 without attention data parallelism. Without the single-server baseline, nobody would have known that disaggregation made things worse.

Two more incidents, both from the router

At 64 concurrent requests, a GLM-4.5 setup logged many "Prefill bootstrap failed" and "Aborted by AbortReq" errors on the prefill side (issue #9266). Four to seven concurrent requests worked. The router's timeout for the prefill client was too short, so the router aborted requests that were still running. Pull request #9803 fixed the timeout.

A maintainer gave the general rule in a related thread (issue #9926). "Aborted by AbortReq" means that the client or the router aborted. Check timeouts before you suspect the transfer.

Novita AI found a different delay with GLM-4.7 on H200 field Novita 2026-01. With disaggregation and the overlap scheduler, throughput rose about 10 percent but time to first token got much worse. On prefill, the KV transfer waited until the next batch's kernel launch. Without CUDA graphs, that launch took hundreds of milliseconds for their 92-layer model. Their fix added a flag, --disaggregation-async-transfer. We could not find that flag in the v0.5.22 code, so check your version before you copy their command.

How many prefill servers for each decode server

The ratio depends on your traffic. dstack tested gpt-oss-120b on eight H200 GPUs with SGLang disaggregation and tensor parallelism only field dstack 2025-09. At high concurrency, one prefill server for three decode servers won every metric, for prefill-heavy, decode-heavy, and balanced loads. At low concurrency, three to one and two to two gave a better time to first token. Their post shows charts without absolute numbers, so use it for the direction only.

At the largest scale, the SGLang team ran DeepSeek on 12 nodes of eight H100 GPUs, with 4 prefill nodes and 9 decode nodes team 2025-05. They reported 52.3k input and 22.3k output tokens per second per node for 2000-token inputs. The Mooncake team reported Kimi K2 on 128 H200 GPUs with 4 prefill and 12 decode nodes team 2025-07. They gave decode more nodes so that the KV pool could hold a batch of 480. Both are reports from the teams that built the system.

What to measure

Measure TTFT and inter-token latency on one server first, with the benchmark in section 5.1. If inter-token latency rises when long prompts arrive, disaggregation can help. After the split, watch these metrics code:

• sglang:num_bootstrap_failed_reqs_total and sglang:num_transfer_failed_reqs_total, which must stay flat
• sglang:kv_transfer_latency_ms and sglang:kv_transfer_speed_gb_s, which show the cost of each move
• sglang:num_decode_prealloc_queue_reqs and sglang:num_decode_transfer_queue_reqs, which show requests that wait for memory or for a transfer.

Sources:python/sglang/srt/arg_groups/fields/disagg.py, python/sglang/srt/arg_groups/choices.py, python/sglang/srt/environ.py, python/sglang/srt/disaggregation/prefill.py, python/sglang/srt/disaggregation/decode.py, python/sglang/srt/disaggregation/base/conn.py (KVPoll), python/sglang/srt/disaggregation/common/conn.py, python/sglang/srt/disaggregation/mooncake/conn.py, python/sglang/srt/managers/data_parallel_controller.py, python/sglang/srt/observability/metrics_collector.py, sgl-model-gateway/src/routers/http/pd_router.rs and pd_types.rs, and docs/docs/advanced_features/pd_disaggregation.mdx, all at v0.5.22. Issues #12591, #9266, #9926, pull request #9803. Novita AI on Hugging Face, 2026-01-22. dstack, "Benchmarking Prefill-Decode ratios", 2025-09-25. LMSYS, 96 H100 deployment, 2025-05-05. The Mooncake team on the LMSYS blog, 2025-07-20.

You start a second server to double capacity. Throughput rises, but time to first token gets worse for long conversations. Each server has its own KV cache, and turn 5 of a chat goes to the server that never saw turns 1 to 4. That server prefills the whole history again. Section 3.4 measured what a prefix miss costs on one server. A router that ignores prefixes makes the same miss on purpose.

When you finish this section, you can explain how sgl-model-gateway picks a server for each request. You can also choose a routing policy for your traffic and check that the gateway keeps your hit rate.

Two places where requests get routed

SGLang routes requests in two places, with different defaults.

Inside one server: with --dp-size above 1, a data parallel controller gives requests to the copies. Its default method, auto, resolves to round_robin outside disaggregation code. Round robin ignores prefixes, as figure 6.1b shows.

In front of servers: sgl-model-gateway is a separate Rust program that routes to many SGLang servers. It was named sgl-router until pull request #14312 renamed it on 2025-12-05, for "less confusion". Its default policy is cache_aware code.

A field report shows the difference. Jarvislabs wrote that "SGLang's default routing deals each request to the next GPU in line, blind to where a conversation's cache lives" field Jarvislabs 2026-08. On their traffic, 23 percent of input tokens were computed again. One request header that pinned each conversation to one GPU raised their cache hit rate from 64.7 to 92.1 percent and halved median TTFT. Their post does not name the component that did the round robin. The in-server default matches their description.

How cache_aware picks a server

The gateway keeps one approximate radix tree per model. Each tree node records which servers have seen that prefix. The tree stores characters of the request text, not tokens, so the gateway never runs a tokenizer to route code.

Match rate: the number of characters of a new request that match the tree, divided by the request length.

For each request, select_worker does three things code:

1. It checks balance. The system is imbalanced when two tests pass. The busiest server's load is more than 64 above the lightest, and more than 1.5 times the lightest. Then the request goes to the least-loaded server.
2. Otherwise it computes the match rate. Above 0.3, the request goes to the server with the longest match.
3. Below 0.3, the request goes to the least-loaded server, with a random choice among ties.

The gateway then inserts the request text under the chosen server. Every 120 seconds, a background task removes least recently used leaves until each tree is at most 67,108,864 nodes code.

A comment at the top of the policy file says that a low match goes to "the worker with smallest tree size". The code sends it to the least-loaded worker. The code wins, and this manual describes the code.

The tree is a guess

The gateway never asks a server what it caches. Its tree records where it sent text, and a server can evict that prefix later. So the gateway can send a request to a server that no longer holds its prefix. That request then prefills like any other miss. Check the hit rate on the servers, with sglang:cache_hit_rate from section 5.2, and not only the routing decisions.

Settings that matter

All defaults come from the gateway source in the v0.5.22 release code. That source says version 0.3.2. The newest gateway tag in the repository, gateway-v0.3.1, points at a commit from 2026-01-08. Pin the gateway by the commit you run, not by that tag.

The source also has bucket and consistent_hashing policies. The command line does not accept them at v0.5.22. For two plain servers, the start command is python -m sglang_router.launch_router --worker-urls http://127.0.0.1:30000 http://127.0.0.1:30001 --policy cache_aware.

Our position

Use cache_aware for chat and agent traffic, where turns share long prefixes. Use power_of_two or round_robin for batch traffic with unique prompts, because a match rate under 0.3 gives no affinity anyway. Do not put multi-turn traffic behind in-server --dp-size with its round robin default. Run separate servers behind the gateway, or set a different --load-balance-method.

The SGLang team introduced the cache-aware router in v0.4 team 2024-12. They reported up to 1.9 times more throughput and a 3.8 times better hit rate. Those are their numbers on their workload. Measure your own hit rate before and after.

Pin the router with the engine

A user reported a two times throughput regression in SGLang on GB200 NVL72 with DeepSeek-R1 (issue #22095). The regression was in the router outside SGLang. With the same SGLang build, Dynamo router 0.8.1 gave 132,140 tokens per second and Dynamo router 1.0.0 gave 69,767. Benchmark the router and the engine as one system, and pin both versions.

A second incident shows that routing and service discovery interact. With disaggregation and Kubernetes service discovery, cache_aware failed with "Failed to select prefill/decode worker", and users switched to round robin (issue #10608). The issue was closed, and we could not find the fix that closed it. Test your exact deployment mode before you depend on a policy.

What to measure

Compare sglang:cache_hit_rate on every server before and after you change the policy. Watch the gateway's metrics on port 29000 for retries and open circuits. The gateway retries a failed request up to 5 times by default, and it opens a server's circuit after 10 failures code. A rising TTFT with a falling hit rate means that requests no longer reach the server that holds their prefix. For a broader view of gateways in production, see the lesson on AI gateways.

Sources:sgl-model-gateway/src/main.rs, sgl-model-gateway/src/policies/cache_aware.rs, sgl-model-gateway/Cargo.toml, python/sglang/srt/arg_groups/serving_hook.py (handle_load_balance_method), and python/sglang/srt/managers/data_parallel_controller.py, all at v0.5.22. Pull request #14312, issues #22095 and #10608. Jarvislabs, "Managed Endpoints", 2026-08-26. LMSYS, "SGLang v0.4", 2024-12-04.

The first production incident with a new serving engine is rarely about tokens per second. A rolling update kills ten requests in the middle of their answers. A new pod takes four minutes to become ready, so the autoscaler starts three more. A liveness probe times out under load and restarts the busiest pod. None of these appear in a benchmark, and all of them cost more than a slow kernel.

When you finish this section, you can choose an image tag and keep startup short. You can connect the health probes to the right endpoints and stop a server without losing work. You can also change settings in the order that this manual measured, and read three production reports for what each one teaches.

Pin the image you run

The SGLang team publishes images on Docker Hub as lmsysorg/sglang. At v0.5.22 the latest tag holds the newest stable release, and latest-runtime holds a smaller image of the same release docs. Nightly tags follow main. The images contain CUDA 13 only. The docs name lmsysorg/sglang:v0.5.19-cu129 as the last CUDA 12 tag, so a host with an older driver needs that tag or a driver upgrade.

The documented start command gives the container a large shared memory segment and the host IPC namespace:

docker run -it --gpus all \
  --shm-size 32g \
  -p 30000:30000 \
  -v "$HF_CACHE_DIR:/root/.cache/huggingface" \
  --ipc=host \
  lmsysorg/sglang:latest bash

Our position: never deploy latest. SGLang tagged v0.5.20, v0.5.21, and v0.5.22 within four weeks, from 2026-09-18 to 2026-10-13, and each release changed defaults that this manual measures. A pinned tag such as v0.5.22 makes a regression a choice that you made. The tag latest makes it something that happened to you on a restart.

Cold start is weights plus compilers

A new SGLang process loads the weights, compiles or loads kernels, sizes the KV pool, and captures graphs. Our capture logs each phase on one line. This run used a model that was already on disk:

[2026-10-16 13:58:06] Engine startup timings (s): load_weight=0.00, kv_cache_allocation=0.00, scheduler_e2e=7.01, cuda_graph={prefill=0.00, decode=0.00, target_verify=0.00, draft_prefill=0.00, draft_decode=0.00, draft_extend=0.00}, tokenizer_e2e=11.92

We measured 18 s from the first log line to "The server is fired up and ready to roll!" for a 0.6B model on the Mac. A large model on GPUs is a different problem. A field report from Doubleword, an engineer blog dated 2026-04, timed SGLang v0.5.10 with Qwen3.5-122B-A10B-FP8 on 8 B200 GPUs field Doubleword 2026-04. A cold start took 695 s, and 531 s of that was weight loading. With kernel caches kept between runs, a warm start took 88 s. With a process checkpoint and weights staged in host memory, it took 9.6 s.

The code helps with the cheap part of that work. When Python imports sglang, it points the Triton, TorchInductor, CUDA, TileLang, and FlashInfer caches into one directory, SGLANG_CACHE_DIR, which defaults to ~/.cache/sglang code. Mount that directory and the Hugging Face cache on volumes that survive the pod. Then a restart reuses compiled kernels and downloaded weights.

The same field report defends the slow start: "SGLang getting 10% more tokens per second is worth way more than launching 10s faster." field Doubleword 2026-04 We agree for a fleet that runs for days. For a fleet that scales to zero every night, startup time is a real cost, and the persistent caches are the first fix.

Health checks: /ready, /health, and /health_generate

SGLang has two kinds of health endpoint code:

• /ready returns 200 when the tokenizer manager is ready and 503 before that.
• /health and /health_generate share one handler. They return 503 while the server starts and while it shuts down.
• Otherwise the handler sends a one-token request with max_new_tokens 1 and waits for any output from the detokenizer.
• SGLANG_ENABLE_HEALTH_ENDPOINT_GENERATION defaults to true, so a plain /health also generates that token.
• The wait is SGLANG_HEALTH_CHECK_TIMEOUT, 20 s by default. With no output in that time, the handler marks the server UnHealthy and returns 503.

The docstring says that the probe request costs nothing when the server is busy, because any output from the scheduler counts as proof of life. That is true for the work, and false for the timing. A real incident: issue #22511 reports "health check failed. server could not get a response from detokenizer for last 20 seconds" on GLM-5 with 2 H800 GPUs under high load, on v0.5.10 issue #22511. A related issue, #35884, found that a timed-out probe left its request in the scheduler, so failed probes accumulated issue #35884.

Our position: use /ready for the readiness probe. Keep the liveness probe from generating. Set SGLANG_ENABLE_HEALTH_ENDPOINT_GENERATION=false, and /health then returns 200 without a request to the scheduler code. Use /health_generate for a slower deep check that alerts a person and restarts nothing. A busy server then does not lose its busiest pod to a probe that waited in the queue.

Graceful shutdown: signal the main process only

On SIGTERM, the HTTP server sets gracefully_exit, and /health starts to return 503. A watchdog in the tokenizer manager checks the flag every 5 s code. It then logs the remaining requests every 5 s until none remain. Then it sends ShutdownReq to the schedulers and gives them 15 s to exit before it kills them. Figure 7.1 shows this path and the two paths that skip the drain.

We measured both cases on the Mac. Our capture kit stopped each server by sending SIGTERM to its whole process group. The scheduler child received the signal too and died first:

[2026-10-16 14:01:45] SIGTERM received. signum=None frame=None. Draining requests and shutting down...
[2026-10-16 14:01:45] Subprocess scheduler_0 (pid=5909) crashed with exit code -15. Triggering SIGQUIT for cleanup...
[2026-10-16 14:01:45] SIGQUIT received. signum=None, frame=None. It usually means one child failed.
[2026-10-16 14:01:46] Sleeping 5 seconds before crash diagnostics to let GPU activity settle.

The parent saw a crashed child, took the crash path, and killed the process tree. No request was running at that moment, so nothing was lost. Under load, every running request would have failed. We then started a server, sent it a 3,000-token request, and sent SIGTERM to the main process only:

[2026-10-16 14:13:54] SIGTERM received. signum=None frame=None. Draining requests and shutting down...
…
[2026-10-16 14:13:55] INFO:     127.0.0.1:53237 - "GET /health HTTP/1.1" 503 Service Unavailable
…
[2026-10-16 14:13:57] Gracefully exiting... Remaining number of requests 1. Remaining requests remaining_rids=['ef12359aaaaf40d78fedb4e7f7aaed25'].
…
[2026-10-16 14:14:04] INFO:     127.0.0.1:53209 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:14:07] Gracefully exiting... Remaining number of requests 0. Remaining requests remaining_rids=[].
…
[2026-10-16 14:14:07] INFO:     Finished server process [53853]

health before SIGTERM: 200
sent SIGTERM to the server process only
health during drain: 503
in-flight request: {"status": 200, "completion_tokens": 3000, "finish_reason": {"type": "length", "length": 3000}, "seconds": 14.2}
server exit code: 0

The request finished with all 3,000 tokens, and /health returned 503 during the drain. The server exited with code 0, thirteen seconds after the signal. docker stop and Kubernetes send SIGTERM to the main process of the container, which is the correct target. A wrapper script that forwards the signal to a process group, or a supervisor that kills the group, gives you the first log instead.

The code has two exits that skip the drain. If the server is already UnHealthy, the watchdog exits at once. It also exits at once if the environment variable SGL_FORCE_SHUTDOWN is true. At v0.5.22 the watchdog reads that old name directly code.

The environment module declares the new name SGLANG_FORCE_SHUTDOWN. Its compatibility code copies SGL_ variables to SGLANG_ names, never the other way. The code shows that setting only SGLANG_FORCE_SHUTDOWN does not reach the watchdog. We read this in the code and did not run it, so we call it a likely bug. If you need a forced stop, set SGL_FORCE_SHUTDOWN=1.

Our position: set the termination grace period longer than your longest request at your output cap. A 32K-token answer at 30 tokens per second needs more than 18 minutes. Cap max_tokens before you set the grace period, or a drain can never finish in time.

The order to change settings

Most tuning advice lists flags. The order matters more than the list, because each step changes what the next step measures. Figure 7.1b shows the order this manual uses, and each row links to the section that measures it.

1. Measure with sglang.benchmark.serving at three concurrency levels, as in section 5.1. Without a baseline, no later change has a number.
2. Set --mem-fraction-static for the machine you share, as in section 3.2. On our Mac the automatic value crashed the server at concurrency 32.
3. Put shared text first in every prompt, as in section 3.4. One request id at the start of a system prompt cost us about 14 times the time to first token.
4. Cap --max-running-requests at the knee you measured. Past the knee, our throughput fell while time to first token grew.
5. Only now change CUDA graph settings and --attention-backend, one change per run, as in section 4.1 and section 4.2. At v0.5.22 --disable-cuda-graph is deprecated in favour of --cuda-graph-backend-decode and --cuda-graph-backend-prefill.
6. Add tensor, data, or attention-data parallelism when one replica is saturated, as in section 6.1.
7. Split prefill from decode last, and only after you measured that prefill delays decode, as in section 6.3.

What production teams report

Three reports by teams outside SGLang show where operating effort goes.

LinkedIn, an independent engineering blog dated 2026-02-20, serves ranking models of 375M to 0.6B parameters on H100 GPUs field LinkedIn 2026-02. The workload needs only prefill and the logits of the last token. Their bottlenecks were the tokenizer, ZMQ messages, Python garbage collection, the GIL, and the scheduler, and none of them were GPU kernels.

Batch tokenization took P99 latency from 4,583 ms to 464 ms. A scoring path that skips decode took P99 from 6,220 ms to 454 ms. Garbage collection pauses of 100 to 300 ms went away with a warm-up and gc.freeze(). The 375M ranker went from 750 to 2,200 items per second per GPU. LinkedIn wrote most of the pull requests involved, so their "before" is an older SGLang.

CERIT-SC, a Czech academic service, moved from vLLM to SGLang in 2025 and reported 100 billion tokens served by 2026-04 field CERIT-SC 2026-04. Their traffic shape is the lesson. For Kimi K2.6 the median prompt was 2,179 tokens and the 95th percentile was 99,746 tokens, which they attribute to agentic coding tools. Their SGLang problems were tool calls that leaked into reasoning output, rare hangs, and model loops that held KV cache. Their fix was operational: an output cap of 32K tokens and a presence penalty for workloads that loop. They still run both engines, chosen per model.

Ant Group published a DeepSeek-R1 deployment on H20 GPUs with SGLang v0.5.2 on the LMSYS blog, dated 2025-09-26 field Ant Group 2025-09. They chose expert parallelism of 16 over 32 because at batch 32 EP16 gave 675 tokens per second against 585 for EP32. More speculative draft tokens gained 21% at batch 1 and lost about 18% at batch 32. They sell three latency tiers, and the strictest tier, below 30 ms TPOT, runs at batch 12. The measured tier table is a product decision, which is the point.

Our position: read a production report for its traffic shape and its order of fixes, never for its throughput number. LinkedIn fixed CPU work before GPU work. CERIT-SC capped output before it tuned anything. Ant Group chose the smaller expert group because the fault radius and the batch size it sells mattered more than peak throughput.

Sources:python/sglang/srt/entrypoints/http_server.py (ready, health_generate, HEALTH_CHECK_TIMEOUT), python/sglang/srt/managers/tokenizer_manager.py (sigterm_watchdog, _SCHEDULER_EXIT_TIMEOUT_SECS), python/sglang/srt/environ.py (SGLANG_FORCE_SHUTDOWN, SGLANG_ENABLE_HEALTH_ENDPOINT_GENERATION, SGLANG_CACHE_DIR, redirect_third_party_caches, _handle_deprecated_envs) and python/sglang/__init__.py at v0.5.22; docs/docs/get-started/install.mdx at v0.5.22; capture/out/server-default.log, capture/out/server-default.command.txt, capture/out/shutdown-exploratory.log, capture/out/shutdown-exploratory.txt, capture/shutdown_probe.py; issues #22511 and #35884; field reports: Doubleword, fergusfinn.com/blog/fast-sglang-starts (2026-04-06), LinkedIn Engineering, linkedin.com/blog/engineering/ai/scaling-llm-based-ranking-systems-with-sglang-at-linkedin (2026-02-20), CERIT-SC, blog.cerit.io/blog/run-llm2 (2025-10-19) and blog-dev.cerit.io/blog/llm3 (2026-04-27), Ant Group on the LMSYS blog, lmsys.org/blog/2025-09-26-sglang-ant-group (2025-09-26).

You train a model with GRPO. Every step, an inference server writes thousands of answers, a reward function scores them, and the trainer updates the weights. Then the server must answer the next prompts with the new weights, on the same GPUs that the trainer just used. A serving engine built for chat never had to give its memory back or change its weights while it runs. It never had to match another program's numbers to the last bit.

When you finish this section, you can describe one RL step with SGLang as the rollout engine and name the SGLang endpoints each phase calls. You can also choose a weight transfer method and say what "true on-policy" covers and what it does not. The example is Miles v0.1.2 (commit a22fd824), the RL framework that RadixArk maintains next to SGLang.

The receipt for training on stale numbers

Miles is a fork of slime, an RL framework from THUDM, and keeps slime's history. RadixArk describes it as "Enterprise-Grade Reinforcement Learning for Large-Scale Model Post-Training". The docs never define "enterprise". In the code it means three things: correctness checks between rollout and training, recovery from engine faults, and observability.

Each of those exists because RL changes what an inference server is. The server's output is the training data. The log probabilities it reports are the "old policy" in the PPO and GRPO ratio. So any numerical difference between SGLang kernels and the trainer's kernels becomes a bias in the gradient. Miles is the receipt for training on numbers that the server did not produce.

The loop that borrows the server

The training driver is short on purpose. At v0.1.2, train.py has 175 lines. These are the lines of one step when the trainer and SGLang share GPUs:

for rollout_id in range(args.start_rollout_id, args.num_rollout):
    await inference_controller.prepare_rollout(rollout_id)
    rollout_data_pack = await rollout_executor.get(rollout_id)

    if args.offload_rollout:
        ...
            offload_tags = [GPU_MEMORY_TYPE_CUDA_GRAPH]
            if "kv_cache" in args.offload_rollout_level:
                offload_tags.append(GPU_MEMORY_TYPE_KV_CACHE)
            if "weight" in args.offload_rollout_level:
                offload_tags.append(GPU_MEMORY_TYPE_WEIGHTS)
            await inference_controller.offload(tags=offload_tags)
    ...
        await actor_model.train(rollout_id, rollout_data_pack)
    ...
            await offload_train()
            if args.offload_rollout:
                await inference_controller.onload_weights()
        await update_weights(args, actor_model, rollout_executor, inference_controller, rollout_id=rollout_id)
        if args.offload_rollout:
            await inference_controller.onload_kv()

Read it as a rental. SGLang generates and scores the rollout. SGLang releases its CUDA graphs, KV cache, and weights. The trainer loads, trains, and offloads itself. SGLang gets its weights back, receives the new weights, and then gets its KV cache and CUDA graphs back. Figure 7.2b shows which memory is on the GPU in each phase.

What stock SGLang gives the trainer

The rental works through SGLang HTTP endpoints that exist in stock v0.5.22 code. SGLang's RL guide states the design rule as "Be a library, not a framework" docs. Miles drives each engine as an ordinary sglang serve process over HTTP.

Memory release: POST /release_memory_occupation frees GPU memory by tag, and POST /resume_memory_occupation gives it back. The server needs --enable-memory-saver, which is off by default. The tags are kv_cache, weights, and cuda_graph. The docs list only the first two, and the code accepts all three, so the code wins. The memory saver frees physical memory but keeps the virtual addresses, so captured CUDA graphs stay valid without a new capture. The call asserts that the server is idle, so pause generation first.

Pause and resume: POST /pause_generation takes a mode. SGLang's default is abort, which ends every running request. Miles uses retract by default, which puts running requests back in the waiting queue so they prefill again after the update. A third mode, in_place, keeps their KV cache from the old weights.

The weight-update session: begin_weight_update, then one or more update_weights_from_* calls, then end_weight_update. The session exists because quantized layers need a one-time repack after loading. Doing it once per refit is cheaper than once per bucket. After the session, update_weight_version stamps the new version. A request that lived through a refit reports weight_versions spans in its meta_info, so you can see which tokens came from which weights. Figure 7.2 shows one update with the default transfer.

Four ways to move the weights

Miles picks the transfer with --update-weight-transfer-mode code:

With --colocate, Miles ignores this flag and always sends CUDA IPC handles through /update_weights_from_tensor. A team that sets p2p on a colocated run measures the wrong thing and never knows.

The Miles team reports P2P timings on H100 clusters with a 1 GB bucket team 2026-10. Qwen3-235B-A22B on 8 nodes went from 10,754 ms with NCCL broadcast to 3,162 ms, 71% less. Kimi-K2 on 32 nodes went from 53,279 ms to 7,227 ms, 86% less. On one node P2P was slower: GLM-4.7-9B-Flash took 68.6% longer and a 4-layer GLM-5 test took 72.2% longer. The reason is in the design. Broadcast sends the full model to every engine rank, and with expert parallelism each rank needs only a fraction of it.

RadixArk reports that about 2% of BF16 parameters change in one step team 2026-08. For GLM-4.7-Flash the disk-delta payload fell from 62.4 GB to 0.69 to 0.83 GB per update, with a generation pause of 3 to 5 s. These are their numbers from the Miles v0.1 post of 2026-08-18. We did not reproduce them.

Our position: start colocated on one node with the CUDA IPC path. Move to separate GPUs only when rollout time dominates the step. Choose p2p only for multi-node MoE, where the vendor table shows the gain, and keep broadcast everywhere else.

True on-policy, and where it stops

Here the serving engine and the trainer must agree on a number. Give SGLang and Megatron or FSDP the same weights and the same tokens. Their kernels can still return different log probabilities, because their reductions run in a different order. Then the PPO ratio mixes "the policy changed" with "the kernels differ". Miles reports the gap as train/train_rollout_logprob_abs_diff, which should be exactly 0 in true on-policy mode.

Two SGLang facts matter first. SGLang returns log probabilities after temperature scaling unless the request is greedy or SGLANG_RETURN_ORIGINAL_LOGPROB is set, as section 4.6 explains. A trainer that compares at temperature 1 sees a mismatch that no kernel causes. Second, --rl-on-policy-target accepts one value at v0.5.22, fsdp, and it forces deterministic inference code. That mode replaces matrix multiply, softmax, and norm kernels with batch-invariant versions and forces the PyTorch sampler. It also turns off the radix cache unless the attention backend is fa3, fa4, triton, or ascend.

Miles builds on that with a launch plan that sets matching flags on both sides. The coverage is narrow. At v0.1.2 the only model profile is Qwen3 dense: Qwen3-0.6B, Qwen3-4B, Qwen3-4B-Base, and Qwen3-4B-Instruct-2507. Three open issues in the Miles repository show where bitwise parity still breaks:

• Issue #1485: Qwen3-4B down_proj outputs differ at tensor parallel 2. SGLang sums 38 partial results in two groups of 19, and Miles sums them in a pairwise tree. BF16 addition is not associative issue radixark/miles#1485.
• Issue #3714: deterministic serving pins the NCCL channel limit to 8 while the trainer uses 24. Weight transfer then fails with Message truncated : received 6144 bytes instead of 2048 issue radixark/miles#3714.
• Issue #3277: after a retract, re-prefill under the new weights overwrites the recorded MoE routes. The response then carries new routes with old log probabilities issue radixark/miles#3277.

Our position: treat true on-policy as a diagnostic for Qwen3 dense models, not a switch for every run. Log the log-probability gap on every run anyway. A large gap means a conversion or kernel problem, and no RL algorithm fixes that.

Miles runs on its own SGLang

Miles does not use the SGLang release that the rest of this manual describes. Its Docker image builds on lmsysorg/sglang:v0.5.21 and checks out the sglang-miles branch. The v0.1.2 release lock pins SGLang commit b84bec5761a6. The Miles docs say it plainly: "installing stock sglang or upstream Megatron-LM next to Miles does not work." docs

At least four things that Miles calls do not exist at v0.5.22: the /pull_weights endpoint for disk-delta, register_lora_adapter, get_parallelism_info, and the --true-on-policy-contract flag.

That branch is a fair summary of this section. RL asks a serving engine for things that serving users never asked for. It must give memory back and change weights in a session. It must report which weights produced each token and match another program bit for bit. Some of those requests are now in stock SGLang. The rest live on a branch until they are ready.

A real incident: issue #860 in the Miles repository reports that /release_memory_occupation "silently does nothing" for models with sliding-window attention issue radixark/miles#860. The sliding-window KV pool turns the memory saver off in its constructor, so a colocated run runs out of memory when Megatron starts. One user saw 30.19 GB of KV pool stay resident on DeepSeek-V4-Flash. The issue was open on 2026-10-16. Check that every memory tag really frees memory on a new model before you trust colocation.

Sources:Miles at v0.1.2 (a22fd824): train.py, release-lock.json, docker/Dockerfile, docs/developer/versions.md, miles/ray/rollout/inference_controller.py, miles/backends/training_utils/weight_update/protocol.py, session.py, protocols/broadcast.py, protocols/cuda_ipc.py, protocols/p2p.py, protocols/delta.py, miles/true_on_policy/model_profiles.py, docs/advanced/p2p-weight-transfer.md; SGLang at v0.5.22: python/sglang/srt/entrypoints/http_server.py, python/sglang/srt/managers/io_struct.py (PauseGenerationReqInput), python/sglang/srt/managers/scheduler_components/weight_updater.py, python/sglang/srt/constants.py, python/sglang/srt/arg_groups/fields/exec_.py, python/sglang/srt/arg_groups/choices.py, python/sglang/srt/arg_groups/attention_hook.py, python/sglang/srt/layers/logprob_processor.py, docs/docs/advanced_features/sglang_for_rl.mdx; Miles issues #860, #1485, #3277, #3714; RadixArk and LMSYS posts: lmsys.org/blog/2025-11-19-miles and lmsys.org/blog/2026-08-18-miles-v0-1.

sglang serve --help lists 535 options at 0.5.22 (capture/out/serve-help.txt). Most of them never matter to you. The table below holds the ones this manual uses. Many defaults are None in the code. A startup hook replaces them with a value that depends on the GPU, the model, and the free memory. The "On our Mac" column shows that resolved value, from the server_args line of capture/out/server-default.log.

Memory and admission

Scheduling

Cache

Execution

--disable-cuda-graph still works at 0.5.22, and the per-phase flags --cuda-graph-backend-decode and --cuda-graph-backend-prefill replace it.

Parallelism and operations

Sources:python/sglang/srt/arg_groups/fields/schedule.py, memory.py, exec_.py, model.py, parallel.py, spec.py, serving.py, observability.py, device.py at v0.5.22; capture/out/serve-help.txt, server-default.log, server-small-pool.log.

--enable-metrics turns on /metrics. Without the flag, the endpoint does not exist, and the default is off. Our default server exported 81 metric families with the prefix sglang: (capture/out/metrics.txt). Section 5.2 explains which ones to alert on. This table keeps the ones the manual uses.

Latency

Load and memory

Cache and tokens

What our runs recorded

After the shared-prefix benchmark, the default server reported sglang:cache_hit_rate 0.94, and the server with --disable-radix-cache reported 0.0 for the same benchmark (capture/out/metrics.txt, no-radix-cache-metrics.txt). The default server counted 286,102 prompt tokens, and 182,079 of them came from the cache. Each counter appears once per label set, so sum the rows before you divide.

Sources:python/sglang/srt/observability/ at v0.5.22; capture/out/metrics.txt, no-radix-cache-metrics.txt, small-pool-metrics.txt.

The sections use real incidents as evidence. Each one comes from an SGLang or Miles issue, a pull request, a field report, or our own capture. This index lists them by part, so you can search it with a symptom you see in your own logs. A status of open or closed was checked on 2026-10-16.

Our capture

Part 1 and Part 2: the request path

Part 3: the KV cache

Part 4: each forward pass

Part 5: measuring

Part 6 and Part 7: fleets and operations

Sources:the incident paragraphs of every section under sections/; capture/out/oom-exploratory.log, server-no-overlap.log, server-default.log; the GitHub issues and pull requests linked in the tables.

Every term is defined once, in the section the last column names. The meanings come from the SGLang source at the tag, the captures, and the papers that the sections cite.

Sources:the term definitions in every file under sections/; front.md.

The ranks match the front matter. When two sources disagree, the higher one wins and the section says so. The SGLang source files each section used are named in the Sources line at its end, at the tag, and are not repeated here. Every page below was read on 2026-10-16.

The ranked sources

Rank 1 and rank 2: the code and the captures

Rank 3: pull requests

The issues are in the incident index. These pull requests explain a design change.

Rank 4: papers

Rank 5: SGLang documentation at the tag

Rank 6: independent field reports

Rank 7: SGLang team and vendor reports