Scenario
More load gave a faster first token
RolesInference EngineeringAI Evaluation and Reliability
- Engine
- SGLang 0.5.22
- Model
- Qwen/Qwen3-30B-A3B-Instruct-2507
- Hardware
- one GB10 (DGX Spark)
Situation
A benchmark kit starts one SGLang server with Qwen/Qwen3-30B-A3B-Instruct-2507 and runs a load sweep against it. Each run sends random prompts of 512 tokens and asks for 128 tokens back. The kit runs each concurrency cap once, one after the other, on the same server.
The report says that a cap of 8 gave a median first token of 677.61 ms, and a cap of 32 gave 390.38 ms. Four times the load gave a faster first token. A teammate wants to publish the result as "the scheduler gets more efficient under load". You review the run before it goes out.
Questions
- Which explanations fit a median first token that falls when the load rises? Name at least two before you read the logs.
- The mean and the P99 at a cap of 32 are much higher than the median. What does that shape tell you?
- What in the client arguments makes two runs send the same prompts?
- What do the prefill lines prove, and what do they not prove?
- How do you run the sweep again so that the result is trustworthy?
Commit to a cause after each listing. Write it down before you read the next one.
Evidence
The server command
The command that the benchmark kit used to start the server. Note what is on and what is not.
sglang serve --model-path Qwen/Qwen3-30B-A3B-Instruct-2507 --host 127.0.0.1 --port 30000 --enable-cache-report --enable-metrics --mem-fraction-static 0.7Source: manuals/sglang-101/capture/gpu/out-gb10/server-default.command.txt
The first-token block at a cap of 8
The first-token block of the report for 64 requests at a cap of 8.
---------------Time to First Token----------------
Mean TTFT (ms): 681.92
Median TTFT (ms): 677.61
P90 TTFT (ms): 1189.25
P95 TTFT (ms): 1194.16
P99 TTFT (ms): 1194.93 Source: manuals/sglang-101/capture/gpu/out-gb10/bench-random-c8.txt, lines 34-39
The first-token block at a cap of 32
The first-token block of the report for 128 requests at a cap of 32. Compare the median with the cap of 8.
---------------Time to First Token----------------
Mean TTFT (ms): 1166.51
Median TTFT (ms): 390.38
P90 TTFT (ms): 2887.53
P95 TTFT (ms): 2889.80
P99 TTFT (ms): 2910.22 Source: manuals/sglang-101/capture/gpu/out-gb10/bench-random-c32.txt, lines 34-39
The client arguments
Look for the dataset, the seed and the cache flush.
benchmark_args=Namespace(backend='sglang', base_url=None, host='0.0.0.0', port=30000, ready_check_timeout_sec=60, dataset_name='random', dataset_path='~/sglang-lab/hf/hub/datasets--anon8231489123--ShareGPT_Vicuna_unfiltered/snapshots/192ab2185289094fc556ec8ce5ce1e8e587154ca/ShareGPT_V3_unfiltered_cleaned_split.json', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model=None, served_model_name=None, tokenizer=None, num_prompts=128, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=512, random_output_len=128, random_range_ratio=1.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=32, output_file='~/sglang-lab/kit/capture/gpu/out-gb10/bench-random-c32.jsonl', output_details=False, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=7, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=False, flush_cache_timeout=60.0, warmup_requests=1, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None, rust_client=False, rust_worker_threads=None)Source: manuals/sglang-101/capture/gpu/out-gb10/bench-random-c32.txt, lines 1-1
Two prefill lines from the run at a cap of 32
Compare the new tokens with the cached tokens in each batch.
[2026-10-17 14:23:00] Prefill batch, #new-seq: 14, #new-token: 14, #cached-token: 6879, token usage: 0.07, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 108.68
[2026-10-17 14:23:00] Prefill batch, #new-seq: 15, #new-token: 15, #cached-token: 7634, token usage: 0.07, #running-req: 15, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 113.27Source: manuals/sglang-101/capture/gpu/out-gb10/server-default.log, lines 341-342
The same caps with three seeds
The single kit run against a sweep with seeds 1, 2 and 3 on the same machine.
| Cap | Single kit run, median TTFT | Three seeds, median TTFT (lowest to highest) |
|---|---|---|
| 8 | 677.61 ms | 209.98 ms (185.25 to 216.02) |
| 32 | 390.38 ms | 416.37 ms (338.06 to 434.64) |Source: manuals/sglang-101/sections/5-1-sglang-benchmark-serving.md, lines 144-147
One load four times in a row
A smaller model on another GPU. Read the ratios, not the times.
engine fresh repeat, not flushed repeat after a flush repeat, not flushed
SGLang 0.6B 394 82.5 289 77
vLLM 0.6B 925 148 1,163 184Source: manuals/sglang-101/capture/gpu/out-l4/prefix-cache-flush.txt, lines 9-11
The answer
Show the walkthrough
- Doubt one sample. No queue model makes the first token faster under more load. Each cap ran once, so the first suspect is the method, not the scheduler. Write down the candidate causes: a warm cache, a different GPU clock, a different pool, or noise in one run.
- Read the whole distribution. At a cap of 32 the median was 390.38 ms, but the mean was 1166.51 ms and P99 was 2910.22 ms. A median far below the mean says the requests split into a fast group and a slow group. Something made part of the requests cheap.
- Read the client arguments. The run used the random dataset with
seed=7andflush_cache=False. The same seed sends the same first prompts in every run. Nothing cleared the cache between runs on the same server. - Read the server log. Two prefill batches admitted 14 and 15 requests that needed one new token each, with 6879 and 7634 cached tokens. Those prompts were already in the cache from the run before. A cached prompt skips its prefill, so its first token comes fast.
- Count, then test. The manual counted the prefill lines of each run. At a cap of 32, 64 of the 128 prompts were in the cache before the run started. At a cap of 8, 16 of the 64 were. Half of the requests had the advantage at 32, and a quarter at 8. Then change the variable that the reading blames: run each cap with three seeds, and the 677.61 ms result does not come back.
Show the resolution
The prefix cache made the result. The kit reused one seed and one server and never flushed the cache, so each run found the prompts of the run before it. The sweep with seeds 1, 2 and 3 gave a median of 209.98 ms at a cap of 8 and 416.37 ms at a cap of 32. With clean runs, the first token gets slower as the cap grows, as the queue predicts.
The fix is a protocol, not a server flag:
- Start each run with an empty cache. Pass
--flush-cacheto the benchmark client, callPOST /flush_cache, or restart the server. - Check that the flush ran. The server flushes only when no request runs or waits.
- Use a new seed for each run, so that the client sends new prompts.
- Record
sglang:cached_tokens_totalbefore and after each run, and write the cached tokens next to the result.
A separate test on an L4 measured the size of the effect. A repeat without a flush read a median first token about 4.8 times lower than the fresh run. After a flush, it went back near the fresh run.
Show the wrong turns
“The scheduler batches better at a high cap.”
A queue can only add wait time. The median fell because the cache removed prefill work.
“Turn the prefix cache off for the benchmark.”
Production runs with the cache on. Reset it between runs and record the hits, so that the test measures the real server.
“Run more requests at each cap with the same seed.”
More requests with the same seed repeat the same prompts and keep the cache warm.
“Blame the GPU clock.”
A clock change can move a result, but the prefill lines show cached tokens, which a clock change cannot make.
Show the general rule
A benchmark that repeats its inputs measures its own cache. Before you trust a result that runs against the expected direction, read the full distribution, check which state carries over between runs, and change the one variable that your explanation blames. Run three seeds per point and report the range. The time per token is less exposed to this error than the first token, because a cache hit removes prefill work and leaves the decode work the same.
Taught in
The benchmark section of SGLang 101 finds this result and its cause, and the load testing lesson explains why identical prompts in a loop test only one point.