Manuals / SGLang 101 / Part 3 · The KV Cache
3.6

Hybrid models and the cache

Sliding-window and Mamba layers keep state that a token-level tree cannot split at any position, so hybrid models are where many reported cache failures sit at v0.5.22.

A team serves a hybrid model that mixes linear-attention layers with full-attention layers. They turn on HiCache and run twenty requests that hit the host tier. All twenty answers are fluent and on topic for a different prompt. The usage block reports full cached tokens, and nothing in the logs warns them.

When you finish this section, you can say why hybrid models break the simple picture of the radix tree. You can also test the cache on your model before your users test it for you.

Three kinds of layer state

The radix tree in section 3.3 works because full attention stores one key and one value per token. A cached prefix of any length is valid, so the tree can split a node at any token. Two other kinds of layer break that rule.

Sliding-window attention (SWA): attention that reads only the last few thousand tokens. A SWA layer needs only the window, so SGLang keeps a smaller pool for SWA layers and drops KV that leaves the window. A cached prefix is useful only where the window's KV still exists.

Mamba layer: a state-space or linear-attention layer that keeps one fixed-size recurrent state per sequence, updated token by token. The state at token 2,000 summarizes all 2,000 tokens. It cannot be cut back to token 1,500. A prefix can resume only where SGLang saved a snapshot of the state.

Our capture model has neither. The startup log says so.

a model with only full-attention layerstextFull log
[2026-10-16 13:58:06] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=False hybrid_ssm=False hicache_attached=False
…
[2026-10-16 13:58:08] Attention KV pool initialized: pool_size=181574 (buffer size 181575 incl. padding slot 0), 28 full-attention layers (0 sliding-window layers stay per-request), 8 kv_heads, 128 head_dim
Two lines from the startup log of the default configuration.

Read these two lines on your own model first. hybrid_swa=True or hybrid_ssm=True means that everything in this section applies to you.

How SGLang handles each kind

UnifiedRadixCache keeps one tree with a component for each kind of state. Each node holds data for the full-attention component and, when needed, for the SWA and Mamba components. A match advances only while every component agrees that the state at that position is valid.

For SWA, the tree keeps a marker where window KV was dropped, and a match stops at the marker. --swa-full-tokens-ratio sets the size of the SWA pool against the full pool. It defaults to None, and model-specific overrides set it. The DeepSeek-V4 override sets it to 0.1.

For Mamba, the tree stores state snapshots on some nodes and copies a snapshot when a request reuses it. --mamba-radix-cache-strategy chooses how SGLang keeps snapshots: auto by default, no_buffer, extra_buffer, or extra_buffer_lazy. --mamba-track-interval sets how often decode saves a snapshot, every 256 tokens by default.

Fig. 3.11a Mamba prefix resumes only at a snapshottree
A Mamba prefix resumes only at a snapshot A cached conversation drawn as a token ribbon from 0 to 2,048 tokens, with Mamba state snapshots every 256 tokens. A new request matches the first 1,800 tokens. Full-attention KV exists for all 1,800 tokens, but the recurrent state exists only at snapshots, so the request resumes from the snapshot at 1,792 tokens and prefills tokens 1,792 to 1,800 again. CACHED CONVERSATION · 2,048 TOKENS 0 512 1,024 1,536 2,048 teal ticks: Mamba state snapshots, every 256 tokens NEW REQUEST · MATCHES 1,800 TOKENS reused: 1,792 tokens, up to the last snapshot prefilled again: 8 tokens from 1,792 to 1,800 Full-attention KV exists for every token. The recurrent state exists only where a snapshot was saved.
A new request that matches 1,800 tokens of a Mamba model resumes from the last snapshot at 1,792 tokens, and prefills 8 tokens. Read from the root down. Snapshots every 256 decode tokens follow the --mamba-track-interval default. The token counts are an example, not a capture.

What goes wrong, in the issues

Many reported cache bugs at v0.5.22 involve hybrid models. We did not run a hybrid model ourselves. These four issues show the range.

  1. A wrong answer from a host-tier hit. Issue #39830 is the opening story of this section. A hybrid linear-attention model with HiCache gave fluent answers to a different prompt on 20 of 20 host-tier hits. Cold requests and GPU hits were correct.
  2. Zero reuse for an identical prompt. In issue #22935, the match of a repeated prompt ends inside an edge, and the node split creates a node with no Mamba state. The match stops at the last node with state, so the identical prompt reuses nothing.
  3. A strategy that makes the first token 4.7 times slower. In issue #37834, an option forced the no_buffer strategy. The snapshot sat at a depth that later requests could not reach. On a retrieval workload the hit rate fell from 88.2 to 65.9 percent. The mean time to first token rose from 226 ms to 1,063 ms.
  4. A default that slowed decode. Issue #36131 reports a new default that frees SWA slots outside the window. It reduced single-stream decode throughput for hybrid SWA models on DGX Spark and Thor.

Our position: test the cache on a hybrid model before you trust it

On a full-attention model, a wrong cache hit is rare enough that we leave the cache on and move on. On a hybrid model, run three checks before production. Each one takes minutes.

  1. Correctness. Send the same prompts cold, with a GPU hit, and, if you use HiCache, with a host-tier hit. Compare the outputs at temperature 0. Run --kv-canary log during the test, which checks KV slots for corruption and keeps the server running.
  2. Reuse. Repeat a prompt and read #cached-token. An identical prompt that shows a small count has hit the split problem of issue #22935.
  3. Latency. Measure the time to first token with --mamba-radix-cache-strategy auto and with the strategy your deployment guide recommends. Keep the faster one only if the hit rate holds.

Then read the open issues for your model family before each upgrade. Between 2026-09-01 and the v0.5.22 tag, about 330 commits touched the cache code in python/sglang/srt/mem_cache. On hybrid models, a new release can fix one of these bugs and add another.

SGLang 0.5.22 · baa9bef · 2026-10-13 · Edition 2026.10