Lab notes

Use this page to check a number from another section, or to repeat a run.

Method

• Hardware and versions. The version pins page lists them.
• Client. Every run used the SGLang 0.5.22 benchmark client. The vLLM runs used --backend vllm, which sends each request to /v1/completions. The client sends one warmup request first, and it asks the server to ignore the end-of-sequence token unless you pass --disable-ignore-eos. Its random dataset cuts or repeats real chat text to the asked length. The benchmark page prints the full command.
• Waves. The waves workload sends 64 prompts at 16 users. The length ratio sets how far the prompt and output lengths can vary. At 1.0 every request has 512 prompt and 128 output tokens. At 0 each length is drawn at random from 1 token up to that size.
• Seeds. The runs to 2026-10-21 used seeds 1, 2, and 3. Most runs on 2026-10-22 used seeds 1 and 2, and the tables give the count. Tables give the median, and some columns give the lowest and the highest.
• Server settings. The sweep tool starts every vLLM server with VLLM_USE_FLASHINFER_SAMPLER=0, because the FlashInfer sampler fails to build on our L4. It also sets --seed 0, --max-model-len, --num-gpu-blocks-override, and --gpu-memory-utilization. The L4 used 8,192 tokens, 1,506 blocks, and 0.9. The GB10 used 8,192 tokens, 14,685 blocks, and 0.75. The shared L4 GPU used 2,048 or 2,304 tokens, 183 blocks, and 0.83. The small-pool run used 4,096 tokens and 410 blocks.
• Rounding. The tables and the sections use one rule. They round the decimal value from the summary file, and a half goes up, so 1,012.5 shows as 1,013.
• Pools. On the L4, vLLM ran with --num-gpu-blocks-override 1506, a pool of 24,096 tokens, against 24,094 for SGLang. On the GB10, vLLM ran with 14,685 blocks, a pool of 234,960 tokens. The SGLang pools on the GB10 were 233,303 tokens for load, 234,958 for waves, and 251,910 for mixed.
• Clocks. A sampler recorded the SM clock and the temperature every second. The sweep tool can wait for the GPU to cool before a run. It can also repeat a run when the median clock falls below a clock floor, a lowest clock that we set for each sweep. The clock column shows the result for each point. On 2026-10-22, most runs had the SM clock locked at 900 MHz with nvidia-smi -lgc. A lock sets the highest clock, but a GPU at a high temperature still slows below it. The sections compare two points only at similar clocks, and they say so where a point ran below the lock or in a throttled session.
• Dates. The L4 load runs of vllm and sglang were on 2026-10-17, and the vllm-b, vllm-async, vllm-eager, and vllm-seqs-16 load runs on 2026-10-21. The L4 rate and waves runs were on 2026-10-18. The L4 prefix runs with the cache on ran from 2026-10-17 into 2026-10-18, and the run without the prefix cache was on 2026-10-21. Every GB10 run in these tables was on 2026-10-21. Every run in the three L4 tables for 2026-10-22 was on that day. The runs on the shared GPU of the L4 node had a pool of 2,928 tokens, so they tested 1 and 4 users only.

• The cache between user counts. The load tables before 2026-10-23 kept one server for each config and did not reset the prefix cache between user counts. Each seed sent the same first prompts at every count. Below 32 users these tables read the first token too fast, by 3.6 to 5.3 times for vLLM on our L4. The rerun of 2026-10-23 resets the cache before every run, and the benchmark page compares the two.
• Runs of 2026-10-23. The L4 runs of that day used the first GPU with the clock locked at 900 MHz. Each run had two seeds and a cache reset first. The card ran hot, so many runs fell below the lock. Each evidence file gives the median clock of its rows, and the sections compare only rows at 900 MHz.

Not run yet

These runs are planned and are not in these tables:

• --max-num-batched-tokens at 512, 2048, and 16384 on mixed prompts on the L4. The GB10 has 2048 and 16384.
• -O1 on the L4.
• a backend comparison with one KV data type.

Tables

The config vllm-async passes --async-scheduling. For these models with no speculation, that is the default, so vllm-async is a second run of the baseline setting on both machines.

L4, load: users at once, 512 prompt and 128 output tokens

L4, rate: requests each second, 512 prompt and 128 output tokens

L4, waves: 64 prompts at 16 users, by length ratio (1.0 means equal lengths)

L4, prefix: shared prefix tokens, 4 groups of 16 prompts, 8 users

L4, throttled session: 512 prompt and 128 output tokens, 2026-10-22

The GPU ran far below its full clock in this session. Compare these rows only with each other, at matching clocks.

L4, clock locked at 900 MHz: 512 prompt and 128 output tokens, 2026-10-22

The lock sets the highest clock. A GPU at a high temperature still slows below it, as the 1-user runs of vllm-lock-b and vllm-lock-small-pool show.

L4, shared GPU with a pool of 2,928 tokens: 512 prompt and 128 output tokens, 2026-10-22

GB10, load

GB10, mixed: prompts of 1,024 to 4,096 tokens, 64 to 256 output tokens, by rate

GB10, waves