Manuals / SGLang 101 / Part 3 · The KV Cache
3.4

Prefix-friendly prompts

The prefix cache matches tokens from the first one, so one changing token near the start of a prompt can cost more than server tuning recovers.

A team runs DeepSeek-V4 for long agent sessions on eight GPUs, with hierarchical caching turned on. Their dashboards show #cached-token: 0 on requests that repeat 100,000 tokens of history. They tune the cache tiers and the write policy. The hit rate barely moves. Then they find a random hash that their relay put into the system prompt of every request.

When you finish this section, you can predict which prompts hit the prefix cache before you send them. You can also prove the hit rate with cached_tokens, the scheduler log, and /metrics.

The experiment: one identifier, three places

The capture kit sends one 2,999-token system prompt and one short question, five times in each of three forms. The server runs with --enable-cache-report, so each response reports its cached tokens to the client. The kit sends two warm-up requests first.

  • Stable: the same system prompt every time.
  • ID at start: the system prompt starts with Request id <random hex>., a new value on every request.
  • ID at end: the same random ID goes at the end of the system prompt.
the summary of the cache experimentjsonFull log
…
{"summary": "stable-system-prompt", "median_ttft_ms": 45.1, "min_ttft_ms": 34.0, "max_ttft_ms": 62.5, "median_cached_tokens": 2998.0, "prompt_tokens": 2999}
{"summary": "id-at-start", "median_ttft_ms": 1255.3, "min_ttft_ms": 1102.6, "max_ttft_ms": 2197.3, "median_cached_tokens": 5, "prompt_tokens": 3034}
{"summary": "id-at-end", "median_ttft_ms": 88.2, "min_ttft_ms": 83.7, "max_ttft_ms": 118.1, "median_cached_tokens": 2985, "prompt_tokens": 3028}
The three summary lines. The fifteen per-request lines above them are cut. The stable case drops its first round, which filled the cache.

We measured a 28 times longer wait for the first token when the ID sits at the start: 1,255 ms against 45 ms. The run used our Mac with Qwen3-0.6B in 4-bit MLX. The factor depends on the prefill speed of the card. The number of missed tokens does not.

The same ID at the end cost 43 ms more than the stable prompt. The tokens are almost the same in all three cases. Only their order differs.

The cached-token counts explain the times. The stable prompt reused 2,998 of 2,999 tokens, the maximum from section 3.3. The ID at the start matched only the first 5 tokens, the ones that come before the random hex digits. Every token after a change is a different branch of the tree, even when the rest of the prompt is identical.

Fig. 3.8the same tokens in three orderscomparison
The same tokens in three orders Three token ribbons for prompts of about 3,000 tokens. The stable prompt reuses 2,998 tokens and computes 1, with a median first-token time of 45 ms. With a random ID at the start, only 5 tokens are reused and about 3,029 are computed, at 1,255 ms. With the same ID at the end, 2,985 tokens are reused and about 45 are computed, at 88 ms. reused from the radix cache computed in prefill STABLE SYSTEM PROMPT 2,998 cached median TTFT 45 ms RANDOM ID AT THE START 5 cached median TTFT 1,255 ms RANDOM ID AT THE END 2,985 cached median TTFT 88 ms Every token after the first difference is a new branch of the tree.
A random ID at the start of the prompt made about 3,027 tokens miss the cache, and the same ID at the end cost about 45 tokens. Read each ribbon left to right as the prompt's tokens. Green is reused from the cache, and plum is computed in prefill.

Why the order matters

The radix tree compares tokens from the first one onward, and it stops at the first difference. Nothing after a difference can match, because the attention values of every later token depend on every earlier token. A cache that reused the later tokens would return wrong answers. So prefix caching is exact by design, and it is unforgiving by the same design.

The same rule applies to each part of a chat request. The chat template turns the messages into one token sequence in a fixed order: system prompt, tool list, earlier turns, new message. A change in the system prompt invalidates everything. A change in the newest message invalidates only that message.

The incident

Issue #35129 is the story at the top of this section. The reporter found three causes stacked on top of each other.

  1. The relay put a per-session random cch= hash and a rotating system message into each request. Removing them moved the engine hit rate from 66.8 percent to 94.4 percent.
  2. The client wrote its task-list snapshot at the end of the system prompt. Every task update changed the system prompt, so it invalidated more than 100,000 tokens of history after it.
  3. Hierarchical cache load-back, when it could not find room, evicted the prefixes of other sessions. The reporter counted 250,000 evict calls against 376 completed load-backs.

The first fix was a prompt edit. No server flag in section 3.5 could have recovered the 27.6 points that the hash cost.

Our position: fix prompts before you tune the server

Treat the prompt layout as part of the serving system. Before you change any cache flag, check these rules against your real requests.

  • Put content that never changes first: the system prompt, then the tool list in a fixed order.
  • Put content that changes per session after that, and content that changes per request last.
  • Never put timestamps, request IDs, session hashes, or random sampling seeds in the system prompt. Send them as request metadata, or at the end.
  • Keep tool and function definitions in a stable order. A JSON object whose keys change order is a different token sequence.
  • Append to the conversation. Do not rewrite earlier turns, for example to summarize them, unless you accept a full prefill.

Two server-side settings break reuse in ways that look like prompt problems. A request for prompt log probabilities from position k can reuse only the first k tokens. Different LoRA adapters never share tree nodes, even for identical tokens, because the adapter is part of the tree key.

Routing can break reuse too. With several servers, a request that goes to a server without its prefix misses the cache, whatever the prompt looks like. Jarvislabs reported that routing each conversation to the same server moved their hit rate from 64.7 percent to 92.1 percent. Section 6.4 shows how the gateway routes by prefix.

How to prove a hit

Use three measurements, from the closest to the request to the most general.

  1. In the response. Start the server with --enable-cache-report, which defaults to off. The usage block then reports cached_tokens. The native /generate API returns cached_tokens in meta_info without the flag.
  2. In the scheduler log. Every Prefill batch line reports #new-token and #cached-token.
  3. In the metrics. /metrics exports sglang:cache_hit_rate, a gauge of recent prefill batches. After the shared-prefix benchmark in section 3.3, our server reported 0.94. The server with --disable-radix-cache reported 0.0.
the scheduler's view of the three prompt formstextFull log
[2026-10-16 13:58:12] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 2998, token usage: 0.02, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 7.09
…
[2026-10-16 13:58:14] Prefill batch, #new-seq: 1, #new-token: 3031, #cached-token: 3, token usage: 0.02, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 1937.91
…
[2026-10-16 13:58:20] Prefill batch, #new-seq: 1, #new-token: 45, #cached-token: 2983, token usage: 0.02, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 225.19
One line for each form, in the order the kit sent them. The other twelve lines are cut.

The lessons on prompt caching and on caching and cost show the same rule for hosted model APIs. Those APIs bill cached input tokens at a lower price.

SGLang 0.5.22 · baa9bef · 2026-10-13 · Edition 2026.10