Manuals / vLLM 101 / Part 5 · Measure
5.3

Decide between vLLM and SGLang

With a fixed number of users, the engines tied on throughput on our machines, and the shape of the traffic decided latency.

You are choosing a serving engine and you want a measurement, not a reputation. When you finish this section, you can match your traffic to one of our side-by-side runs and say how far the result goes.

Our result: on our L4, throughput matched within 3 percent up to 32 users. On our DGX Spark, the worst token gap of vLLM was about twice that of SGLang from 32 users.

Know what we compared

We ran vLLM 0.31.0 and SGLang 0.5.22 with their defaults on two machines:

  • An NVIDIA L4 with Qwen/Qwen3-8B in BF16, in the full-clock session. Both pools held about 24,100 tokens. The load runs were on 2026-10-17, and the rate and waves runs on 2026-10-18.
  • A DGX Spark with Qwen/Qwen3-30B-A3B-Instruct-2507 in BF16. The pools held 234,960 tokens for vLLM and between 233,303 and 251,910 tokens for SGLang. Each SGLang run was on the same day as its vLLM run, 2026-10-21.

Each point is the median of three seeds, from one benchmark client for both engines. This is two models on two cards. Treat the results as a guide to what you should measure, not as a rule.

Read the result for your traffic

Your trafficWhat we measuredPick
Many users at once, near the pool limitThroughput tied within 3 percent up to 32 users on the L4. vLLM gave the first token sooner, SGLang gave a smoother stream.vLLM for the first token, SGLang for the stream
A fixed request rate past what the GPU can serveAt 4 requests per second, vLLM gave 11 percent more output on the L4vLLM
Prompts of mixed length, 16 users at oncevLLM had the lower P99 ITL on both machinesvLLM by default
Prompts of the same length, 16 users at onceSGLang had the lower P99 ITL on both machinesSGLang
Long prompts of mixed length at a fixed rate, DGX SparkvLLM had the lower P99 ITL at 0.5 requests per second, SGLang at 2measure at your rate
Repeated prefixes that fit the poolBoth caches gave the first token within about 100 ms of each othereither one
A mixture-of-experts model on a DGX Spark, many usersThroughput tied. The vLLM P99 ITL was about twice the SGLang value from 32 users.SGLang for the stream

See the numbers behind each row

Many users at once, L4. At 32 users, vLLM gave 273.9 tok/s and SGLang 271.3. The median first token took 1,813 ms on vLLM and 2,871 ms on SGLang. The P99 ITL was 619 ms on vLLM and 102 ms on SGLang. At 64 users the output differed by 6 percent. The vLLM run had a lower median clock, 1,013 against 1,215 MHz, so we do not count that gap.

These runs did not reset the prefix cache. We ran them again with a reset at 900 MHz on 2026-10-23, and the direction held. At 32 users the median first token was 2.0 to 2.1 s on vLLM and 3.2 s on SGLang. The P99 ITL was 668 to 964 ms on vLLM and 105 to 106 ms on SGLang. At 16 users it was 522 ms against 82 ms.

A fixed rate, L4. At 4 requests per second, vLLM gave 286.6 tok/s and SGLang 257.5. The seed ranges did not overlap. At 8 requests per second the ranges overlapped. The runs at 16 requests per second ran below our clock floor, the lowest clock we accept for a run, so we left them out.

Mixed and equal lengths, both machines. Our waves workload sent 64 prompts at 16 users, with a length ratio that sets how much the prompt and output lengths vary. The lab notes define it. With mixed lengths on the L4, the P99 ITL was 191 ms on vLLM and 363 ms on SGLang. With equal lengths it was 502 ms on vLLM and 81 ms on SGLang. The DGX Spark showed the same direction: 188 against 198 ms with mixed lengths, and 394 against 159 ms with equal lengths.

Mixture of experts, DGX Spark. At 64 users, vLLM gave 239.3 tok/s and SGLang 245.1, inside the seed spread. The P99 ITL was 528 ms on vLLM and 271 ms on SGLang, and 478 against 220 ms at 32 users. The seed ranges did not overlap. The median first token was also longer on vLLM at every load from 2 users.

Long prompts at a low rate, DGX Spark. With prompts of about 1,000 to 4,000 tokens at 0.5 requests per second, vLLM had the lower P99 ITL, 571 against 922 ms. At 1 and 2 requests per second, SGLang had the lower one. For this sweep the SGLang pool was 7 percent larger than the vLLM pool, so read this row with care.

Fig. 5.2throughput on the L4chart
On the L4 the two engines wrote the same tokens per second up to 32 users Output tokens per second against users at the same time on an NVIDIA L4 with Qwen3-8B, the same 24,096-token KV pool, three seeds per point. The two lines are within 3 percent up to 32 users. At 64 users SGLang wrote 256.6 and vLLM 241.4 tokens per second. 0 100 200 300 1 2 4 8 16 32 64 users at the same time output tokens per second vLLM 0.31.0 SGLang 0.5.22
On our L4 with Qwen3-8B, vLLM and SGLang wrote the same output within 3 percent up to 32 users. Both servers had the same KV pool. Bands span three runs.
Fig. 5.3the worst token gap on the GB10chart
On the GB10 the worst token gap of vLLM doubled from 32 users The 99th percentile inter-token latency against users at the same time on a DGX Spark with Qwen3-30B-A3B, three seeds per point. The engines match up to 8 users. At 32 users vLLM shows 478 ms and SGLang 220 ms, and at 64 users 528 ms and 271 ms. 0 200 400 600 1 2 4 8 16 32 64 users at the same time P99 inter-token latency, ms 478 220 vLLM 0.31.0 SGLang 0.5.22
On our DGX Spark, the 99th percentile token gap of vLLM was more than twice that of SGLang at 32 users. A higher line is a choppier stream. Bands span three runs.

Decide

Pick on the things that do not change with tuning first:

  1. Model support. Check that your model and its quantization run on the engine and the version you pin.
  2. Startup time. On our DGX Spark, vLLM took 330 seconds to load the 57 GiB of weights.
  3. Your hardware. A result on an L4 or a GB10 does not carry over to a datacenter GPU without a test.

Then run the sweep on your own traffic with both engines. Use one benchmark client and equal pools.

vLLM 0.31.0 · db9527a · 2026-10-05 · Edition 2026.10