Manuals / vLLM 101 / Part 1 · Start
1.3

Understand what happens to your request

A request spends one short phase reading the prompt and a longer phase writing the answer, and four numbers measure those two phases.

You sent a request and the answer came back, slowly at first and then token by token. When you finish this section, you can name the two phases of a request and the four numbers that every benchmark in this manual reports.

Our result: on our DGX Spark, 64 users got eight times the output of one user. Each user waited about 7.5 times longer for every token.

Two phases

Prefill: the step that reads the whole prompt at once and writes its keys and values into the KV cache. One prefill step handles many prompt tokens, so the GPU is busy with math.

Decode: the steps that write the answer, one token per request per step. Each step reads all the model weights to make one token for each running request. The GPU waits for memory more than for math.

vLLM does not run the two phases as separate stages. Each scheduler step has one token budget. Running requests get their next decode token first. Then the scheduler spends the rest of the budget on prompt tokens, in chunks. A long prompt can take several steps. Part 8 explains why vLLM works this way.

Fig. 1.2one request through three processesflow
One request through the three vLLM processes A client sends HTTP to the API server process, which tokenizes and sends the request over ZeroMQ to the engine core process. The engine core schedules it and sends the batch to a GPU worker, one for each GPU. Token ids come back the same way, and the API server streams text to the client. YOU PROCESS 1 PROCESS 2 ONE FOR EACH GPU client curl, OpenAI SDK, bench API server HTTP routes tokenizer engine core Scheduler KVCacheManager GPU worker GPUModelRunner forward pass 1 2 3 1 HTTP request, text in 2 ZeroMQ message, token ids 3 shared memory queue, one batch per step token ids back, one step at a time text streamed to the client A worker crash shows up in the engine core log. A slow tokenizer shows up as slow first tokens.
A request crosses three processes: the API server tokenizes it, the engine core schedules it, and a GPU worker runs the model. Solid arrows carry the request forward, dashed arrows carry token ids back, and the dotted arrow is the text stream.

Four numbers

NumberWhat it measuresWho feels it
TTFT, time to first tokenthe wait from the request to the first token, mostly queue time and prefilla user who waits for the answer to start
TPOT, time per output tokenthe mean time between tokens after the first onea user who reads a long answer
ITL, inter-token latencyeach gap between two tokens, so its 99th percentile shows the worst stallsa user who sees the stream stop and start
Throughputoutput tokens per second for all users togetherthe person who pays for the GPU

One server, two loads

The numbers trade against each other as users grow. Here is our DGX Spark serving Qwen3-30B-A3B with vLLM 0.31.0, from one user to 64 users at the same time.

Users at the same timeMedian TTFTMedian TPOTOutput tokens per second
1257 ms32.3 ms29.5
64965 ms243.8 ms239.3

With 64 users, the server wrote about eight times as many tokens per second. Each user waited almost four times as long for the first token, and each token took more than seven times as long. A busy server is cheaper per token and slower for each user.

Every tuning page in this manual moves one of these four numbers. Each page says which number it moves and which number it costs.

Fig. 1.3one server, two loadschart
More users, more tokens in total, a slower stream for each user vLLM 0.31.0 on a DGX Spark with Qwen3-30B-A3B, three seeds per point. Left: output tokens per second for all users rise from 29.5 at one user to 239.3 at 64 users. Right: the median time per output token for each user rises from 32.3 ms to 243.8 ms. 0 100 200 300 1 4 16 64 users at the same time output tokens per second, all users 0 100 200 300 1 4 16 64 users at the same time median ms per token, each user median of three runs, band from lowest to highest
On our DGX Spark, 64 users got eight times the total output of one user, and each user waited about 7.5 times longer for every token. Read the left chart for the server and the right chart for one user. Bands span the lowest and highest of three runs.

vLLM 0.31.0 · db9527a · 2026-10-05 · Edition 2026.10