Manuals / vLLM 101 / Part 8 · How vLLM Works
8.2

Why one token budget replaces a prefill phase

This page explains the scheduler loop, decode first, and preemption by recompute.

Most explanations of LLM serving describe a prefill phase and a decode phase. The vLLM scheduler has neither. When you finish this section, you can explain what the scheduler does in one step, and why that design makes chunked prefill the default.

One question for every request

The scheduler asks the same question about every request: how many tokens does this request still need to compute? A new request needs all of its prompt tokens. A running request needs one token, or more with speculative decoding. A comment in the scheduler source says the same in its own words. No decode phase and no prefill phase exist, only a count of tokens still to compute.

In each step, the scheduler has a budget of tokens, set by --max-num-batched-tokens. For vllm serve on a GPU with less than 70 GiB, such as the L4, the default is 2,048. It spends the budget in this order:

  1. It gives each running request the tokens it needs, up to the budget.
  2. It admits waiting requests while budget and room for running requests remain. --max-num-seqs limits the running requests. For vllm serve, the default is 256 on a GPU with less than 70 GiB and 1,024 on a larger one other than an A100.

A long prompt that does not fit the rest of the budget gets a part of the budget now and the rest in later steps. That is chunked prefill. It is on by default, because the budget design needs nothing else to support it.

Fig. 8.2one token budgettimeline
One 2,048-token budget per step: decode first, prompt chunks fill the rest Three scheduler steps with the default budget of 2,048 tokens. Eight running requests take one decode token each, and a new 4,096-token prompt fills the rest in chunks of 2,040 tokens. In step 3 the last 16 prompt tokens run, and the new request decodes from step 4. BUDGET OF EACH STEP, --MAX-NUM-BATCHED-TOKENS 2048 step 1 2,040 prompt tokens of the new request step 2 2,040 prompt tokens of the new request step 3 the last 16 prompt tokens, decode starts in step 4 decode, one token for each running request a chunk of a new prompt
Each step spends 2,048 tokens: running requests get a decode token first, and a new prompt fills the rest in chunks. Each row is one scheduler step. Prompt chunks are drawn to scale, and decode blocks are widened so you can see them. Dashed space is unused budget.

Why decode goes first

Running requests come first, so a user who already sees a stream keeps seeing it. A new prompt waits for the budget that is left. The vLLM documentation states the trade. A smaller budget gives a smoother stream, and a larger one gives a faster first token for long prompts. The small default of 2,048 is a choice for the stream. GPUs with 70 GiB or more get 8,192, and the long prompts page lists the rule.

What happens when memory runs out

A running request needs a new KV block every 16 tokens. If no block is free, the scheduler preempts a request. Under the default first-come policy, it picks the newest running request. It frees all of its blocks and puts the request back at the head of the waiting queue. When the request runs again, vLLM computes its tokens again from the start.

Older vLLM versions could also copy the blocks to CPU memory and back. The current engine only recomputes, because the documentation judges the recompute cheaper in this architecture. The cost shows up as a longer time for the preempted request, and as the vllm:num_preemptions counter.

Why one budget

One budget for every kind of work keeps the scheduler simple. Prefix caching only lowers the count of tokens to compute. Speculative decoding only raises the count for a running request. The same loop serves all of them. The cost is that one number decides the balance between the first token and the stream. No single value of that number suits every workload.

vLLM 0.31.0 · db9527a · 2026-10-05 · Edition 2026.10