Reading other people's SGLang benchmarks

The most repeated SGLang number on the web is "29 percent faster than vLLM": 16,215 tokens per second against 12,553. It comes from one offline test by the analyst firm AIMultiple, updated 2026-01-25. That page lists SGLang as "v0.2.3" next to vLLM 0.11.0, a gap of about two years of releases. At least five later posts repeat the pair, some under the names of other companies. None of them reran the test.

When you finish this section, you can take a published SGLang benchmark apart into its five inputs. You can decide whether it applies to you and name the mechanism behind it. You can also say where the independent tests agree, which is a shorter list than the headlines suggest.

The five inputs behind every result

A benchmark result is a function of five inputs. Most disagreements between published reports trace back to one of them.

1. Workload shape. Input length, output length, concurrency, and the share of the prompt that repeats. The radix cache helps only when prefixes repeat. Long inputs stress prefill. High concurrency stresses the scheduler and the KV pool.
2. Memory budget. SGLang computes --mem-fraction-static from the GPU when you leave it unset. vLLM uses --gpu-memory-utilization 0.9. A test that sets one and not the other compares two different KV pools.
3. CUDA graph ceiling. The decode step replays a captured CUDA graph only up to a batch size limit. Above it, decode runs eager, one kernel launch at a time. Section 4.1 explains the mechanism.
4. Kernel path. The attention backend, the MoE kernel, and the quantization format. Several reports below were decided by a kernel that did or did not exist for one GPU.
5. Version and date. One analyst measured the same model on the same AMD GPU at 20.4 and then 2,256 tokens per second per GPU, 26 days apart.

The code sets the third input from GPU memory at startup. Take an 80 GB H100 with tensor parallel size below 4. The default decode graph limit is 256, and the chunked prefill size is 8,192. On a 32 GB card the limit is 48, raised from 24 on 2026-09-04. The code comment on that change says the old cap made throughput at high concurrency fall far behind vLLM.

Six questions before you trust a number

1. Is the SGLang version stated? Without it, the result describes an unknown build.
2. Does the workload match yours? Unique prompts say nothing about a chat server with a long shared system prompt.
3. Were the memory budgets matched across engines?
4. Did the concurrency pass the decode graph limit? If yes, the test measured eager decode.
5. Are the flags mapped correctly? The same idea has different flag names in each engine.
6. Who paid for the test? A GPU cloud, a hardware vendor, a competitor, or the SGLang team each has a reason to like one answer.

Where independent tests agree

These reports come from people outside the SGLang team. Each row names the mechanism from earlier sections that explains it.

The links: Spheron, Modal, Varad More, Steve Rawlinson, RunPod, Jarvislabs, SemiAnalysis, Crusoe, and InferenceX.

Read the caveats with the numbers. Rawlinson left SGLang memory at its default and vLLM at 0.9, so the two KV pools differed. Jarvislabs did not record either engine version. Spheron, RunPod, Modal, and Jarvislabs sell GPU time, which is a reason to run tests and no reason to favor one engine. The Crusoe result is the only one here that passed an external audit. Its p99 time per token was 79.98 ms against a limit of 80.

Fragile on new hardware

The same reports show one more pattern. A Blackwell workstation card ran SGLang only with --disable-cuda-graph, according to joshua8.ai. A mixed cluster crashed when an older card joined, because the FP4 path required compute capability 10.0. Pivot News summarized that community post. The fast paths are written per GPU generation, and the paths for other GPUs are fewer. On new hardware, expect the first SGLang release to be slow or broken, and measure again every week.

What our own machines showed

Field reports tell you which mechanism to test. We tested them on a DGX Spark with a GB10 and on a machine with two 24 GB L4s. Each row below links to the section with the listings, the seeds, and the chart.

Most rows follow one pattern. A mechanism wins by a large factor when it removes repeated work, in either engine. Examples are the prefill of a shared prompt and the bytes of each cached token. It loses when a feature spends memory that the batch needs, or when two parts of a deployment are not the same size. None of the losses at runtime printed an error that a dashboard would show. Two of them looked like success in the benchmark summary.

SGLang against vLLM on the same L4

Every field report above compares two engines that someone else configured. We ran our own pair on one L4 with Qwen/Qwen3-8B and the closed-loop sweep of section 5.1. We made three choices so that only the engine differed.

• The same KV pool. The vLLM server got --num-gpu-blocks-override 1506, which is 1,506 blocks of 16 tokens. SGLang sized 24,094 tokens for itself on the same card.
• The same card state. Both engines ran in one session on one GPU, and every run started at 72 C or below. A run with a median SM clock under 1,000 MHz was set aside and repeated, up to three times.
• No accidental handicap. The vLLM server got --max-model-len 8192, which every request fits, and VLLM_USE_FLASHINFER_SAMPLER=0, because its FlashInfer sampler did not compile in our install. The benchmark is greedy, so the sampler does not change the output.

[2026-10-17 18:27:40] KV Cache is allocated. dtype: torch.bfloat16, #tokens: 24094, K size: 1.65 GB, V size: 1.65 GB
[2026-10-17 20:12:04] Decode batch, #running-req: 37, #token: 22162, token usage: 0.92, cuda graph: True, gen throughput (token/s): 432.55, #queue-req: 27

INFO 10-17 20:13:17 [kv_cache_utils.py:2464] GPU KV cache size: 24,096 tokens, Maximum concurrency for 8,192 tokens per request: 2.94x
INFO 10-17 22:35:43 [loggers.py:320] Engine 000: Avg prompt throughput: 460.3 tokens/s, Avg generation throughput: 323.6 tokens/s, Running: 40 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 8.7%

We measured a tie in throughput. Up to 32 users, the two engines were within 3 percent of each other, and time per token was within 6 percent. At 64 users the median time per token was 106 ms on SGLang and 129 ms on vLLM. Below the point where the pool fills, SGLang gave the first token sooner, 170 against 253 ms at 8 users. Raw runs

Above that point, the engines made different choices, and the stats lines show both. SGLang ran 37 requests and kept 27 waiting, with the pool 92 percent full. The vLLM server ran 40 requests and kept 11 waiting. It filled 99.5 percent of its pool while it prefilled new prompts at 460 tokens per second.

The vLLM server started more requests, so their first token came sooner. Each new prefill then shared the step with the running decodes, so the streams stalled. At 64 users the P99 inter-token gap was 1,363 ms on vLLM and 114 ms on SGLang.

The clock caveat belongs with the last row. Even after three tries, the vLLM runs at 64 users had a median clock of 1,013 MHz against 1,215 MHz for SGLang. So part of the throughput gap at 64 can be the card. The 12-times gap in inter-token latency is too large for a 17 percent clock difference.

The same pair on shared prompts

We then ran the prefix sweep of section 3.3 on the same pair, with the same pools and the same cool-down rule. Each run sent 64 prompts in 4 groups of 16, at 8 users. The prompts in a group shared a system prompt of 0 to 4,096 tokens, then had a 128-token question and 64 tokens of output. SGLang used its radix cache. The vLLM server used its prefix caching, which is on by default at 0.31.0.

INFO 10-17 23:37:11 [kv_cache_utils.py:2464] GPU KV cache size: 24,096 tokens, Maximum concurrency for 8,192 tokens per request: 2.94x
INFO 10-18 00:43:16 [loggers.py:320] Engine 000: Avg prompt throughput: 231.0 tokens/s, Avg generation throughput: 93.6 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.2%, Prefix cache hit rate: 87.2%

The SGLang log shows the same thing in its own words. In one prefill batch at 4,096 shared tokens, 12,952 tokens came from the cache and 406 were new.

We measured a tie on throughput, within 2 percent at every shared length. The median TTFT of both engines stayed between 354 and 488 ms. At 0 and 256 shared tokens, SGLang gave the first token sooner, outside the seed spread: 354 against 408 ms and 470 against 488 ms. From 1,024 tokens up, the seed ranges overlapped. Raw runs

vLLM leaned ahead on the tail. Its P99 TTFT was lower from 256 shared tokens up, 1,303 against 1,887 ms at 1,024. SGLang had the shorter time per token at 4,096, 75.6 against 80.8 ms. The median SM clock of every run was 1,395 MHz or higher.

The size of the result is in the third line. With the cache off, a separate SGLang run needed 14,706 ms for the median first token at 4,096 shared tokens. That card ran under our clock floor. Both cached runs stayed under 500 ms. The difference between the two caches was about 100 ms.

We then ran the vLLM server with --no-enable-prefix-caching, with the same pool and the same sweep. Its log shows the cache off and every prompt token prefilled again.

INFO 10-21 06:20:24 [kv_cache_utils.py:2464] GPU KV cache size: 24,096 tokens, Maximum concurrency for 8,192 tokens per request: 2.94x
INFO 10-21 08:07:49 [loggers.py:320] Engine 000: Avg prompt throughput: 1607.0 tokens/s, Avg generation throughput: 6.4 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.7%

We measured the same size of gain in both engines. At 4,096 shared tokens, the cache cut the median first token about 31 times on SGLang and 36 times on vLLM. It gave 4.2 times the output on SGLang and 4.4 times on vLLM. Without a cache, the two engines tied on throughput within 7 percent. Raw runs

Without a cache, the vLLM server gave the first token sooner at 256 to 2,048 shared tokens. Its SGLang counterpart ran on another day. The vLLM run with the cache off was on 2026-10-21, four days after the vLLM run with the cache on. The runs at 4,096 shared tokens stayed under our clock floor after three tries, at 930 and 675 MHz. Read that row for its size, not its exact value.

The same pair at a fixed arrival rate

A closed loop sends a new request only when one comes back, so the engine sets the pace. Production traffic does not wait for the engine. We ran the rate sweep of section 5.1 on the same pair, with the same pools and the same rules for heat and clock. Requests arrived at a Poisson rate from 0.5 to 16 per second, with 512 input and 128 output tokens. Each run sent at least 32 prompts, or 30 seconds of arrivals.

[2026-10-18 00:43:57] KV Cache is allocated. dtype: torch.bfloat16, #tokens: 24094, K size: 1.65 GB, V size: 1.65 GB
[2026-10-18 03:13:55] Decode batch, #running-req: 37, #token: 22424, token usage: 0.93, cuda graph: True, gen throughput (token/s): 370.33, #queue-req: 71

INFO 10-18 03:15:35 [kv_cache_utils.py:2464] GPU KV cache size: 24,096 tokens, Maximum concurrency for 8,192 tokens per request: 2.94x
INFO 10-18 06:10:22 [loggers.py:320] Engine 000: Avg prompt throughput: 1326.7 tokens/s, Avg generation throughput: 158.4 tokens/s, Running: 43 reqs, Waiting: 30 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.6%

We measured a tie up to 2 requests per second. Both engines kept up with the arrivals, so throughput was within 1 percent. SGLang gave the first token sooner at 0.5 and 1 per second, outside the seed spread. The medians were 284 against 317 ms and 166 against 180 ms. vLLM had the shorter time per token. It was 68 against 74 ms at 1 per second, outside the spread, and 89 against 100 ms at 2. Raw runs

Goodput shows what that means for a target. We set the target at 1 s to the first token and 100 ms per token. At 2 requests per second, vLLM served a median of 1.36 requests per second within the target, and SGLang 0.88. The seed ranges overlapped, 1.09 to 1.44 against 0.54 to 1.18. The SGLang median time per token was 99.9 ms, on the line of the target.

From 4 requests per second, both engines fell behind the arrivals, and no request met the target. The stats lines show the same admission choice as in the closed loop. SGLang ran 37 requests with the pool 93 percent full. The vLLM server ran 43 requests and filled 99.7 percent of its pool.

At 4 per second, vLLM gave 286.6 against 257.5 tok/s, 11 percent more and outside the seed spread. Its median first token came at 8.3 against 9.6 s, but those seed ranges overlapped. The cost was the stream: its P99 time per token was 187 against 127 ms, outside the spread.

The clock does not explain the gap at 4. The vLLM runs had the lower median clock, 1,245 against 1,350 MHz. At 8 per second the throughput ranges overlapped. At 16 per second, every run of both engines stayed under 1,000 MHz after three tries, so we do not compare that row.

The same pair on waves

The last workload is the one from section 5.1 that makes requests arrive in waves. Each run sent 64 prompts of 512 tokens with 128 tokens of output, at a cap of 16 users. Only --random-range-ratio changed. At 1.0 every prompt has the same length, so requests finish together and their replacements arrive together. At 0.5 and 0.0 the lengths are mixed, so requests arrive one by one.

INFO 10-18 18:38:07 [kv_cache_utils.py:2464] GPU KV cache size: 24,096 tokens, Maximum concurrency for 8,192 tokens per request: 2.94x
INFO 10-18 19:14:22 [loggers.py:320] Engine 000: Avg prompt throughput: 960.0 tokens/s, Avg generation throughput: 187.3 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.6%, Prefix cache hit rate: 8.5%

We measured a different winner for each traffic shape. With fixed lengths, the two engines tied on throughput. They made the same trade as at a full pool. The vLLM server gave the first token sooner, 1,499 against 1,775 ms, outside the seed spread. SGLang kept the stream smooth, with a P99 ITL of 81 against 502 ms. Raw runs

With mixed lengths, vLLM led. At a ratio of 0.5 it gave 169.6 against 159.6 tok/s, 6 percent more and outside the seed spread. Its median time per token was 81.4 against 87.8 ms, and its P99 ITL was 191 against 363 ms. At 0.0 the throughput ranges overlapped, and vLLM still had the lower time per token and the lower P99 ITL. Every median clock was 1,410 MHz or higher.

Our reading: when new prompts arrive one by one, each prefill shares a step with the running decodes. The vLLM server split that work so that the stream stalled less.

To test that reading, we ran SGLang again with --enable-mixed-chunk on 2026-10-21, with the same sweep and the same pool. This flag puts the running decodes into the prefill batch, so a new prompt does not stop the stream. The log shows one prefill of 514 tokens with 15 requests running in the same batch.

[2026-10-21 08:34:19] KV Cache is allocated. dtype: torch.bfloat16, #tokens: 24094, K size: 1.65 GB, V size: 1.65 GB
[2026-10-21 08:39:51] Decode batch, #running-req: 16, #token: 8889, token usage: 0.37, cuda graph: True, gen throughput (token/s): 231.46, #queue-req: 0
[2026-10-21 08:40:22] Prefill batch, #new-seq: 1, #new-token: 514, #cached-token: 0, token usage: 0.34, #running-req: 15, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 2569.89

We measured that the mixed chunk closed the gap. With mixed lengths, SGLang now tied vLLM on throughput, time per token, and P99 ITL, and every seed range overlapped. Against SGLang without the flag, the P99 ITL fell from 363 to 193 ms at 0.5 and from 302 to 196 ms at 0.0. The output rose by about 8 percent. Raw runs

The flag had a cost with fixed lengths. The median P99 ITL rose from 81 to 203 ms, and the three seeds spread from 83 to 349 ms. When a whole wave arrives together, the prefills now share steps with the decodes, so the stream loses the clean gaps that it had before. The vLLM server stayed worse there, at 502 ms.

The GB10 with Qwen3-30B-A3B, against a default run on the same day, moved in the same direction, but every change was inside the seed spread. With mixed lengths, the median P99 ITL went from 198 to 164 ms at 0.5 and from 214 to 202 ms at 0.0. With fixed lengths, it went from 159 to 210 ms. The flag also raised the median TTFT at 0.0 from 324 to 442 ms, outside the spread. Read the GB10 as a weak second sign, not a second proof.

One caveat applies. The SGLang default run on the L4 was from 2026-10-18, three days before the mixed-chunk run. The median clocks were 1,425 to 1,508 MHz with the flag and 1,440 to 1,485 MHz without it. The clock does not explain the gap.

This is one model, one GPU, and four workloads. It agrees with the field reports on the main point. On a dense model with unique prompts, the engines were within 3 percent on throughput in the closed loop up to 32 users. Under load they were up to 11 percent apart.

The sweep adds the part that a throughput chart hides. When the pool is full, you choose between a fast first token and a smooth stream, and the two engines chose differently by default. With mixed prompt lengths, vLLM was ahead by default on throughput and stream smoothness. SGLang still gave the first token at the same time or sooner. One SGLang flag, --enable-mixed-chunk, removed that lead.

To see why one GPU number differs from another, run the Inside an LLM Serving GPU machine. It reads its pool sizes, step times and throughputs from the same summaries as this section, for the L4 and the GB10. It says when a setting was not measured.

Our position

On our L4 with Qwen3-8B and unique prompts, the choice of engine was mostly not a throughput decision. It was a latency decision, and the shape of the traffic picked the winner. When requests arrived together, at a full pool or in waves, SGLang protected the stream and vLLM the first token. When prompts of mixed lengths arrive one by one, vLLM led by default on throughput and stream smoothness. With the mixed chunk on, SGLang tied it.

At 4 requests per second, past the full pool, vLLM also gave 11 percent more output on our L4. At 8 per second the seed ranges overlapped. Decide on model support, startup time, and the hardware you have, then measure your own length mix.

For repeated prefixes, check that the prefix cache is on, then measure. On our L4, with prefixes that fit the pool, the two caches tied under 500 ms. With the cache off, SGLang took 14,706 ms and vLLM 13,353 ms. The cache is the result, not the engine. For DeepSeek-class MoE models on many GPUs, start from SGLang, because that is where the field runs them.

Above all, do not import a result. Run the sweep from section 5.1 on your own workload. A published number tells you which mechanism to test, and nothing more.

The incidents

A memory result that was a flag error. In vLLM discussion #17221, a user first saw SGLang use 7 GB against vLLM's 21 GB on one A10. Then the user found that the flag that matches vLLM's --max-model-len is --context-length, not --max-total-tokens. With the right flag, memory use was about the same.

Advice that no longer parses. Field reports from 2026 advise raising the decode graph limit with --cuda-graph-max-bs. At v0.5.22 that flag is split into --cuda-graph-max-bs-decode and --cuda-graph-max-bs-prefill. The code rejects the old spelling. The argument parser of sglang serve stops with ambiguous option: --cuda-graph-max-bs could match --cuda-graph-max-bs-decode, --cuda-graph-max-bs-prefill. A tuning recipe is tied to its version as tightly as a benchmark is.

The recycled number. The 29 percent pair from the start of this section was criticized by DeepInfra on 2026-08-04 for its version gap. The author answered on 2026-10-14 that the result does not carry over to long context. The SGLang team made a larger claim in 2024: up to 3.1 times the throughput on Llama-70B. No independent report found in this research reproduced a gap of that size on a dense model with unique prompts.