Keep long prompts from stalling short ones

Long prompts arrive and short requests stall behind them. When you finish this section, you can choose a batch token budget for your latency target.

Our result: on our DGX Spark at low load, a budget of 16,384 tokens raised the P99 ITL from 574 to 798 ms.

Know the budget

vLLM always splits long prompts into chunks. In each step, the running streams get their next token first, and a waiting prompt fills the rest of the budget. --max-num-batched-tokens sets the budget. The token budget page explains the order.

A larger budget lets a long prompt finish its prefill in fewer steps, so its first token comes sooner. But each of those steps takes longer, and every running stream waits for it. So at low load, a larger budget can give a faster first token and longer gaps between tokens.

Know your default

vLLM sets the budget and the cap on running requests from the GPU memory that it reads. The defaults also differ between vllm serve and the offline LLM class. Other manual pages link to this table.

Our DGX Spark has 121.69 GiB of memory that the CPU and the GPU share. Its startup log still shows a budget of 2,048, the value for a small or unknown GPU. Read the value that your server uses before you change it. The 0.31.0 log does not print the budget under its own name. Find compile_ranges_endpoints in the engine config line. The largest number in that list is the budget.

See what our runs show

We measured budgets of 2,048 and 16,384 tokens on our DGX Spark with Qwen3-30B-A3B on 2026-10-21. The prompts had 1,024 to 4,096 tokens, and each request asked for 64 to 256 tokens. We sent requests at a fixed rate, with three seeds per point. Raw runs

At 0.5 requests per second, the larger budget raised the P99 ITL by 39 percent, and the seed ranges did not overlap. The first token came sooner, but the seed ranges of the first token overlapped. So the cost is clear, and the gain is not confirmed.

The median first token at 0.5 requests per second was slower than at 1 request per second, for both budgets. With the 2,048 budget, all three seeds at 0.5 were above 900 ms. All three at 1 were below 600 ms. So one slow seed does not explain it. We did not find the cause.

At 1 and 2 requests per second, the two budgets gave the same numbers within the seed ranges. At those rates, more requests share each step, so a single long prompt fills less of the budget.

Read our result as narrow: two budgets, three low rates, one model, and one machine. The 2026 paper "P-PAS: Prefill-Pressure Adaptive Scheduling for Long-Context LLM Serving" found that no single fixed budget in vLLM was best at every load. A large budget gave lower latency at low pressure. A small budget won when several prefills competed with running streams. We did not measure that crossover ourselves.

The run with no budget flag matched the run with 2,048 at every rate. That agrees with the 2,048 in the startup log. All runs held a median clock of 2,398 to 2,411 MHz.

Tune it

1. Read the budget that your server uses.
2. If the P99 ITL is too high and long prompts arrive, lower the budget. Try 2,048 if your default is larger.
3. If the first token of long prompts is too slow and the P99 ITL has room, raise it.
4. Measure at your real request rate, because the effect changed with the rate in our runs.
5. Measure again when your load changes, because the best budget moves with the load.

vLLM also has --long-prefill-token-threshold. It caps the tokens that one long prompt gets in a step when other requests wait. It is off by default, and we did not test it.