What determinism costs a live server

What the mode switches

On vLLM, VLLM_BATCH_INVARIANT=1 needs a GPU of compute capability 8.0 or higher, which includes the L4 and the GB10, and it turns off speed features that change the order of the sums, such as custom all-reduce with tensor parallelism. At 0.31.0 it is in beta.

On SGLang, --enable-deterministic-inference replaces the matrix multiply, the log-softmax, the mean, the RMS norm and the batched multiply with batch-invariant kernels, forces the plain sampling backend, sets the all-reduce to a tree so that tensor-parallel sums add in a fixed order, accepts only some attention backends, turns the radix cache off unless the backend is one that supports it, and gives every request without a sampling_seed the seed 42. The cache rule matters most: with the default backend on the GB10 the log said that the radix cache is not compatible, so every repeated prefix paid its full prefill again. On the L4 the mode picked a different backend that keeps the cache on.

What the manual measured

vLLM on the L4 with the clock locked at 900 MHz, Qwen3-0.6B, runs in the order default, batch-invariant, default again, two seeds each with a cache reset before each run:

On Qwen3-8B at 8 users the time per token was 1.4 times longer and the output 27 percent lower; at 32 users 1.3 times and 23 percent. The relative cost fell as more users shared each step, and it was larger on the small model.

SGLang on the GB10 with Qwen3-30B-A3B, one run per server at concurrency 8: output fell from 78.92 to 64.83 tokens per second, 18 percent, and the median time per token rose from 97.30 to 117.43 ms. The radix cache was off in that run, so a workload with shared prefixes pays more than this number shows.

What other teams measured

A post on nondeterminism ran 1,000 Qwen3-8B requests on vLLM in 26 s by default, 55 s with batch-invariant kernels, and 42 s with a better attention kernel. A 2026 study measured batch-invariant kernels at more than twice the latency and up to 74 percent less throughput. A verification study found that the modes of both engines gave different logits under some changes in execution, such as prompt chunking and KV cache reuse. The manual did not measure these itself.

What the mode does not cover

At 0.31.0 a vLLM model that picks its own attention backend skips the check, so its output can still depend on the batch; the next release checks every attention layer, and a fix for expert models with data, tensor and expert parallelism under compile is not in it. On SGLang a hybrid state-space model was reported as not batch-invariant, some newer architectures are not yet supported, a sliding-window kernel gave several greedy outputs in prefix mode with the issue still open, and one speculative algorithm gave different text after a restart because kernel autotuning picked different tactics on each start.

What I would say

Use the mode where identical bits are the product: reinforcement learning rollouts and evaluations that must reproduce. Keep production traffic on the default kernels, where prefix caching and faster kernels are worth more than identical bits. Measure the cost at the concurrency you serve, not at one point, and test the exact setup: model, attention backend, parallel layout, chunk size and cache setting. Flush the cache between two runs you compare, and keep the dtype the same.