RadixCache

A chat product sends the same 2,000-token system prompt with every question. A server without prefix reuse computes those 2,000 tokens for every request, and every user waits for the same work again. On our Mac, that repeated work made a shared-prefix benchmark take 129.7 seconds. With the radix cache on, the same benchmark took 48.8 seconds.

When you finish this section, you can explain how the tree finds a prefix and why a running request keeps its prefix. You can also say how eviction removes cached tokens. You can also read the scheduler log lines that prove a hit.

The same workload with and without the tree

The capture kit runs SGLang's own shared-prefix benchmark twice. Four groups of 16 prompts share a 2,048-token system prompt per group, and each prompt adds a 128-token question. The second server starts with --disable-radix-cache.

We measured a 2.7 times shorter run and a 5.3 times shorter median wait for the first token. The scheduler log shows where the time went.

[2026-10-16 14:01:00] Prefill batch, #new-seq: 7, #new-token: 955, #cached-token: 15102, token usage: 0.05, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 1472.94

[2026-10-16 14:02:34] Prefill batch, #new-seq: 3, #new-token: 4096, #cached-token: 0, token usage: 0.06, #running-req: 2, #queue-req: 3, #pending-token: 7883, cuda graph: False, input throughput (token/s): 2261.21
[2026-10-16 14:02:36] Prefill batch, #new-seq: 3, #new-token: 4096, #cached-token: 0, token usage: 0.08, #running-req: 4, #queue-req: 1, #pending-token: 3787, cuda graph: False, input throughput (token/s): 2245.89
[2026-10-16 14:02:38] Prefill batch, #new-seq: 2, #new-token: 3787, #cached-token: 0, token usage: 0.10, #running-req: 6, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 1626.02

With the tree, seven new requests needed 955 new tokens, and 15,102 tokens came from the cache. Without the tree, the scheduler filled three batches of up to 4,096 tokens for eight requests, all computed from zero. The 4,096 is the chunked prefill size from section 2.2.

The same test on a 24 GB L4

A Mac makes every prefill slow, so we repeated the test on an NVIDIA L4 with Qwen/Qwen3-8B. This time we changed the length of the shared system prompt from 0 to 4,096 tokens. Each run had four groups of 16 prompts, a 128-token question, 64 output tokens, and a cap of eight concurrent requests. We ran three seeds per length.

… Prefill batch, #new-seq: 3, #new-token: 405, #cached-token: 12952, token usage: 0.58, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1452.28
… Prefill batch, #new-seq: 4, #new-token: 546, #cached-token: 17269, token usage: 0.58, #running-req: 4, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 2532.09

… Prefill batch, #new-seq: 2, #new-token: 4096, #cached-token: 0, token usage: 0.68, #running-req: 1, #queue-req: 0, #pending-token: 5527, cuda graph: True, input throughput (token/s): 1632.42
… Prefill batch, #new-seq: 2, #new-token: 4096, #cached-token: 0, token usage: 0.74, #running-req: 2, #queue-req: 0, #pending-token: 1431, cuda graph: True, input throughput (token/s): 1634.70

In this run the L4 prefilled about 1,600 tokens per second, on a card that was throttling. Each 4,096-token batch cost about 2.5 seconds, and without the tree every request paid for its full prompt.

The median SM clock was 1,440 MHz with the cache and 930 MHz without it at 4,096 shared tokens. Every seed of the run without the cache stayed under our floor of 1,000 MHz.

We measured a 30.7 times shorter median wait and 4.3 times the throughput at 4,096 shared tokens. Treat both factors as upper bounds. The run without the cache had a thermal throttle flag in 62 percent of its samples and a 35 percent lower clock. A cool card would shrink the factor, but a 35 percent clock gap cannot explain a factor of 30. With no shared prompt, the median TTFT differed by 8 percent, inside the seed spread, and the throughput was the same. Raw runs

The same sweep on the GB10 with Qwen/Qwen3-30B-A3B-Instruct-2507 gave a smaller factor. At 4,096 shared tokens, the median TTFT was 589 ms with the tree and 5,208 ms without it, 8.8 times. The output rate was 120.66 against 49.73 tok/s. The server without the tree ran a day after the other one. It ran at a median clock of 2,376 MHz, with a throttle flag in 37 percent of its samples. The server with the tree ran at 2,496 MHz.

Our hypothesis for these two cards is that prefill was cheaper on the GB10. That model activates about 3 billion parameters per token, so each repeated prompt cost less to compute again. Two confounds remain: the throttled L4 run without the cache inflates the L4 factor, and the GB10 pair ran on two days at different clocks.

The tree

RadixCache: a radix tree over token sequences, whose nodes hold the KV slot indices of the tokens on their edges. The design comes from the SGLang paper, which calls the mechanism RadixAttention. A radix tree stores runs of tokens on its edges, so one node can hold a whole system prompt.

TreeNode: one node of the tree. It holds its children, keyed by their first token, and the token run on its edge. It also holds the KV slot indices for those tokens, a lock count, a last-access time, and a hit count.

Four operations do all the work.

1. match_prefix walks from the root and follows the child whose first token matches. If the match ends inside an edge, it splits that node so the boundary is exact. Every node on the path gets a new access time.
2. insert walks the same way and adds a leaf for the tokens that did not match. If another request already inserted the same tokens, SGLang frees the duplicate slots and points the request at the tree's copy.
3. inc_lock_ref and dec_lock_ref count the running requests that use a node. A count above zero protects the whole path to the root.
4. evict frees leaves that no request locks, oldest first, and moves up the branch as leaves disappear.

One rule surprises people. A match can cover at most the prompt length minus one. The last prompt token always goes through prefill, because the model needs its logits to sample the first output token. Our cache experiment shows that rule: a 2,999-token prompt reported 2,998 cached tokens and 1 new token.

Which tree you get at v0.5.22

The class name in old blog posts and issues is RadixCache. At v0.5.22 the default is UnifiedRadixCache, one tree that also handles sliding-window and Mamba state. The tree walks run in a Rust core by default since v0.5.21. The log on our Mac shows the exception.

[2026-10-16 13:58:06] Using the Python TreeCore: the Rust TreeCore supports Linux only
[2026-10-16 13:58:06] Init Unified Radix Cache. Components: (<ComponentType.FULL: 0>,). Tree Core: UnifiedTreeCore
[2026-10-16 13:58:06] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=False hybrid_ssm=False hicache_attached=False

The no-cache server printed the same three lines. --disable-radix-cache still builds UnifiedRadixCache, in a disabled mode that matches nothing and inserts nothing. The separate ChunkCache class that older posts describe was deleted on 2026-10-08.

How the scheduler uses the tree

The default --schedule-policy is fcfs, first come, first served. The paper's longest-prefix-match order, lpm, is a choice you make. The code sorts by prefix length only while 128 or fewer requests wait, because matching every waiting request costs time. Above that, lpm uses the FCFS order.

The scheduler also avoids a waste inside one round. When several waiting requests share a prefix that is not cached yet, it lets one of them prefill the shared part first. The others reuse it in the next round. Section 2.2 covers the policies in full.

Our position: leave the radix cache on

In our runs without a shared prompt, on the L4 and on the GB10, the cache cost nothing measurable. We did not test its cost under heavy eviction. Cached tokens live in the same pool as running requests, and eviction frees them on demand. We know of one reason to turn it off: to test whether a wrong answer comes from reused KV. Even then, try --kv-canary log first, which checks KV slots for corruption while the cache stays on.

Then measure the hit rate together with every other feature you add. Issue #32459 is the warning. A team that replayed multi-turn agent traffic saw a deep-prompt hit rate of 97 percent without speculative decoding, and 40 to 53 percent with EAGLE. Input throughput fell from 43,981 to 12,881 tokens per second, with no error in the logs. The candidate cause was the extra KV that speculative decoding reserves, which forced the tree to evict cached prompts.

Field reports put the folklore in proportion. vLLM also reuses prefixes, so engine comparisons measure two caches against each other. One independent test of 16 models on an A10G found SGLang ahead on repeated prefixes. The gain was 20 to 40 ms of first-token time at 7B to 9B. A RunPod test of a 70B model with a cached 7,000-token context measured 35.0 against 32.8 tokens per second per stream. The SGLang paper reports up to 6.4 times higher throughput, which is the authors' best case against the systems of 2024.