Why one ID at the top of a prompt kills the cache
RolesInference EngineeringLLM Product EngineeringAgent Systems Engineering
The question
Every request to your service starts with the same long system prompt, and the prefix cache is on. The server reports almost no cached tokens, and the first token is slow. Last week a teammate added a request ID to the start of the system prompt for tracing. What happened, and what do you change?
Answer it first, aloud or on paper. Then open the answer.
The answer
Show short answer
The prefix cache matches a prompt from its first token and stops at the first token that differs. A new ID at the start makes every later token a miss, so the server computes the whole prompt again on each request. Move the parts that never change to the front and put the ID after them. Then prove the fix with the cached-token counters.
Show full answer
Why the position matters
The attention values of each token depend on every token before it. So a cache can reuse a token only when the whole prefix before it is the same. The SGLang radix tree compares tokens from the first one and stops at the first difference. vLLM does the same with blocks of 16 tokens. It caches only full blocks and stops at the first block that differs.
What it costs
The SGLang manual measured one system prompt of 2,999 tokens in three layouts:
| Layout | Median TTFT | Median cached tokens |
|---|---|---|
| The same prompt every time | 45.1 ms | 2998.0 |
| A random ID at the start | 1255.3 ms | 5 |
| The same ID at the end | 88.2 ms | 2985 |
With the ID at the start, only the 5 tokens before the random digits matched. The first token took 28 times longer. With the ID at the end, the cost was 43 ms.
The fix
- Put the parts that never change first: the system prompt, the tool definitions and the fixed instructions.
- Put the parts that change rarely next, such as the documents of one session.
- Put the parts that change on every request last: the user question, the time and the request ID.
- Send the trace ID in a header or a request field, not in the prompt text.
Check the chat template too. Some templates put the date into the system prompt.
The proof
On vLLM, read vllm:prefix_cache_hits and vllm:prefix_cache_queries from /metrics. The hits divided by the queries is the hit rate. On SGLang, start the server with --enable-cache-report and read usage.prompt_tokens_details.cached_tokens in each response. A good layout shows almost every prompt token as cached after the first request.
Follow-ups
L2WorkingThe hit rate is still low after the fix. What is the next cause to check?
Show expected answer
a full pool, because the cache keeps old blocks only until a running request needs the memory.
L2WorkingWhere in a chat request does a change cost the least?
Show expected answer
in the newest message, because the template puts it last, so only that message is computed again.
L3SeniorTwo tenants send the same system prompt. What is the risk, and what is the control in vLLM?
Show expected answer
one tenant can learn from timing what the other sent. Send a different
cache_saltfor each tenant, so that their requests never share blocks.L3SeniorA team tunes the cache tiers and the eviction policy, and the hit rate does not move. What do you check first?
Show expected answer
the prompt text, for a value that changes on each request near the start, such as a random hash or a timestamp.
Wrong answers
Show the wrong answers that interviewers hear
“Give the cache more memory.”
A bigger pool keeps more blocks, but a prompt that differs at the first tokens has no block to reuse.
“The cache matches the longest shared part anywhere in the prompt.”
The cache matches only from the first token, so a shared part after a difference never hits.
“Turn on hierarchical caching.”
A second tier stores more prefixes. It cannot match a prefix that is different on every request.
Taught in
The prefix-friendly prompts section measures the three layouts, the vLLM prefix cache section gives the counters, and the RadixAttention lesson builds the radix tree.