Fit more users with an FP8 KV cache

Your server queues requests because the KV cache is full. When you finish this section, you can double the KV pool and check the cost in speed and accuracy.

Our result: on our L4, the first FP8 start failed, and the Triton backend fixed it. With the pool held fixed, one user got 2.3 percent faster per token.

Turn it on

The KV cache stores two values for each layer, head, and token. By default, vLLM stores them in the model data type, BF16 for Qwen/Qwen3-8B, at 2 bytes per value. FP8 stores each value in 1 byte:

vllm serve Qwen/Qwen3-8B --kv-cache-dtype fp8

The weights stay in BF16. Only the KV cache changes.

Count what it gives you

Use the formula from the memory page with 1 byte per value. For Qwen/Qwen3-8B, a token needs 144 KiB in BF16 and 72 KiB in FP8.

The same memory holds twice the tokens. So twice the users fit at the same length, or each user can have a context twice as long.

Fix the start on an L4

On our L4, the first start with --kv-cache-dtype fp8 failed. The cause is the attention backend:

1. vLLM tries FlashAttention first on the L4.
2. FlashAttention in vLLM 0.31.0 reads an FP8 KV cache only on Hopper and Blackwell data center GPUs. The L4 is an Ada GPU, SM 8.9.
3. vLLM moves to FlashInfer, the next backend in its list.
4. FlashInfer builds its kernels on the first start. On our node, that build failed.

(EngineCore pid=814129) INFO 10-22 09:11:03 [cuda.py:526] Using FLASHINFER attention backend out of potential backends: ['FLASHINFER', 'TRITON_ATTN'].
(EngineCore pid=814129) ERROR 10-22 09:11:53 [core.py:1433] /data/vllm/.venv/lib/python3.12/site-packages/flashinfer/data/cccl/libcudacxx/include/cuda/std/__cccl/cuda_toolkit.h:41:8: error: #error "CUDA compiler and CUDA toolkit headers are incompatible, please check your include paths"
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

The FlashInfer sampler fails to build in the same way on this node. So our sweep tool starts vLLM with VLLM_USE_FLASHINFER_SAMPLER=0.

Pick the Triton backend. It reads an FP8 KV cache and does not use the FlashInfer build. Also turn off the FlashInfer sampler, as our sweep tool does:

VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve Qwen/Qwen3-8B --kv-cache-dtype fp8 --attention-backend TRITON_ATTN

The attention backend page explains how vLLM picks a backend.

See what our run shows

We measured the FP8 KV cache with the Triton backend against the BF16 default on our L4 with Qwen/Qwen3-8B on 2026-10-22. Both ran in the throttled session, at 1 and 32 users. Raw runs

We held the pool at 24,096 tokens in both runs, as a fixed pool. So this run does not show the larger pool. It shows only the change in speed.

At one user, FP8 was 2.3 percent faster per token, at the same clock, and the seed ranges did not overlap. At 32 users, the seed ranges overlapped, and the FP8 run had a higher clock. So we do not claim a gain at 32 users.

Two things changed together: the KV data type and the backend. The table does not show which one gave the gain. The GPU was throttled for both runs, so read the ratio, not the times.

Check the accuracy

On 2026-10-23 we ran the first 500 GSM8K questions, 8-shot and greedy, on our L4. Qwen3-8B has no KV scales, so the FP8 KV runs used a scale of 1.0. The FP8 KV runs used the Triton backend, as above.

engine   config                     correct   accuracy   Wilson 95% interval
…
vLLM     BF16                       459/500   91.8%      89.1 to 93.9
vLLM     FP8 weights                453/500   90.6%      87.7 to 92.9
vLLM     FP8 KV cache               454/500   90.8%      87.9 to 93.0
vLLM     FP8 weights + FP8 KV       453/500   90.6%      87.7 to 92.9

Every interval overlaps the BF16 interval, so this run did not show a difference. With 500 questions, it cannot rule out a loss of a few points. That is one model and one benchmark, so run the same check on your model and your task. A model with a different attention layout can lose more.

The post "The State of FP8 KV-Cache and Attention Quantization in vLLM" on the vLLM blog, of April 2026, tested uncalibrated FP8 KV caches. Its authors are from AWS and Red Hat. In their tests, most models stayed close to BF16. Some did not. They name Kimi-K2.5, which uses the FlashMLA backend, as a model that did not work well with uncalibrated scales. We did not measure these results ourselves.

Check that your scales load

A checkpoint with calibrated scales does not always use them. At 0.31.0, issue 54623 reports that the Qwen3.5 model family never loads its KV scales. Issue 60333 reports the same for one MTP draft loader. Both issues are open.

First, look for the scales in the checkpoint. Search the weight index in the Hugging Face cache for tensor names that end in k_scale or v_scale. Change the model folder name to your model:

grep -oE '"[^"]*(k|v)_scale"' ~/.cache/huggingface/hub/models--Qwen--Qwen3-8B/snapshots/*/model.safetensors.index.json | head

No output means that the checkpoint has no scales.

A checkpoint with one weight file has no index. For it, list the tensor names with the safetensors Python package.

Then read the startup log. The warning Using KV cache scaling factor 1.0 means that vLLM uses a scale of 1.0. If the checkpoint has scales and the log still shows the warning, the loader dropped them.

Last, run your own prompts with and without FP8, and compare the answers before you go to production.

Know what changed after 0.31.0

Release 0.32.0 adds an NVFP4 KV cache on SM 8.x and SM 12.x GPUs, which include the L4 and the GB10. It stores fewer bytes per token than FP8. We did not test it.