Manuals / SGLang 101 / Part 1 · Start
1.3

What the server does with your request

Every request has two phases with different costs, and four numbers tell you how a server handles them for all its users.

You sent two requests in section 1.1, and both came back in under a second. With ten users, the same server can make some of them wait seconds for the first word. The reason is in the two phases of every request.

When you finish this section, you can name the two phases of a request and the four numbers a server is judged by. You can also say what SGLang controls and what stays with the model and with you.

Two phases with two different costs

Prefill: the forward pass over the whole prompt. It computes the keys and values of every prompt token in one batch and produces the first output token. Decode: the loop that produces each next token, one forward pass per token per request.

The two phases use the GPU in different ways. Prefill has many tokens in each pass, so it keeps the arithmetic units busy. Decode has one new token per request in each pass.

Each decode step reads the weights that the step needs, and the cached keys and values, to do little arithmetic. For a dense model, that is every weight. For a mixture-of-experts model, it is the shared weights and the experts that the router chose. Memory bandwidth sets the speed of decode, and compute sets the speed of prefill.

We measured the difference on a Mac with a 0.6B model. A 512-token prompt reached its first token in a median of 163 ms. After that, each new token took 4.38 ms on average.

one request at a time, 512 tokens in and 128 tokens outtextFull log
Max request concurrency:                 1         
Successful requests:                     16        
…
Median TTFT (ms):                        162.98    
…
Mean TPOT (ms):                          4.38
Cut from the benchmark summary. The latency percentiles other than the median and the inter-token block are cut.

That is at most about 0.32 ms per prompt token against 4.38 ms per output token, a factor of about 14 on this Mac. The 0.32 ms is an upper bound, because the TTFT also holds the queue and the first decode step. On our L4 with Qwen3-8B the factor was about 160: 189 ms for a 512-token prompt, against 60 ms per output token. A serving engine can share and skip prefill work. It mostly batches decode work, so that one read of the weights serves many requests. Section 4.4 shows the other way to cut decode steps.

Fig. 1.1prefill against decode on the capture Maccomparison
Prefill against decode on the capture Mac Two columns from one benchmark run with one request at a time. On the left, prefill runs one forward pass over 512 prompt tokens, writes 512 KV entries, and returns the first token after a median of 162.98 ms, about 0.32 ms per prompt token. On the right, decode runs one forward pass for each of the next 127 tokens, reads every weight and every cached KV entry each time, and takes a mean of 4.38 ms per token. PREFILL · THE WHOLE PROMPT AT ONCE DECODE · ONE TOKEN PER PASS 512 prompt token ids random dataset, seed 7 one forward pass 512 tokens in one batch of matmuls 512 KV entries written kept for every later step median TTFT 162.98 ms about 0.32 ms for each prompt token 1 new token per request the token decode made last one forward pass reads every weight for one token reads 512 + n KV entries and writes one more mean TPOT 4.38 ms repeated for each of the next 127 tokens Qwen3-0.6B-4bit · SGLang 0.5.22 on MLX
The median TTFT divided by the prompt length is about 0.32 ms per prompt token, and one output token costs 4.38 ms in decode. Read left to right. Both columns come from one benchmark run with one request at a time.

The four numbers

Judge a server by four numbers together. A report that gives one of them tells you little.

  • Time to first token (TTFT): how long a user waits before the first word. Prefill and the queue set it.
  • Time per output token (TPOT): the average gap between tokens after the first. Inter-token latency (ITL) is the same gap, measured one token at a time, so its tail shows stalls. Decode sets both.
  • Throughput: output tokens per second across all users. Batching sets it.
  • Goodput: requests per second that meet a TTFT target and a TPOT target at the same time. Your users feel this number.

The first three trade against each other. A larger batch raises throughput and makes the tokens of each user slower. A large prefill chunk finishes a long prompt sooner and stalls every decode in the batch while it runs. Section 5.1 shows the trade on a real sweep.

The SGLang benchmark tool prints TTFT, TPOT, ITL, and throughput. It does not print goodput at version 0.5.22, so compute it from the records of each request. The inference metrics lesson builds that calculation.

What SGLang owns, and what it does not

SGLang owns the time between a request and its tokens. It decides which prompt tokens to compute, which to reuse, and when each request runs.

Fig. 1.2what SGLang ownslayers
What SGLang owns A stack of nine layers from the client at the top to the hardware at the bottom. SGLang owns the tokenizer manager, the scheduler, the KV pool with its radix cache, the forward pass, and the detokenizer. The model gateway is a separate component. The model weights decide answer quality, and the operator decides the memory budget. LAYER OWNER client curl, the OpenAI SDK, sglang.benchmark.serving yours sgl-model-gateway routes requests across servers, section 6.4 separate TokenizerManager HTTP and OpenAI API, text to token ids SGLang Scheduler batches, admission, retraction, overlap SGLang KV pool and UnifiedRadixCache 181,574 slots, cached prefixes SGLang forward pass attention backend, CUDA graphs, MLX on a Mac SGLang DetokenizerManager token ids to text, stop strings, streaming SGLang model weights mlx-community/Qwen3-0.6B-4bit, answer quality the model memory budget and hardware --mem-fraction-static, other programs operator
SGLang owns tokenizing, scheduling, the KV cache, and the forward pass, while the model and the operator own quality, memory, and routing. Read top to bottom, from the client to the weights. Grey layers are outside SGLang.

Three things stay outside. The model decides what the answer says. On our Mac, a 0.6B model asked to name a planet answered "Ardens", which is no planet. SGLang served that answer correctly and fast, and no serving engine can make a model right.

You, the operator, decide how much memory SGLang gets and which machine a request goes to. SGLang has a separate router for that second job, and section 6.4 covers it.

The path of one request

Inside SGLang, your /generate request crossed three processes. The tokenizer manager turned the text into token ids. The scheduler found two of the five prompt tokens in the cache and computed the other three. The detokenizer turned the new token ids into text.

The scheduler logged its side of the request in one line:

the scheduler's view of the request on our L4textFull log
[2026-10-17 14:43:41] Prefill batch, #new-seq: 1, #new-token: 3, #cached-token: 2, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 2.82
[2026-10-17 14:43:41] INFO:     127.0.0.1:58314 - "POST /generate HTTP/1.1" 200 OK
The prefill line and the response line of the /generate request from section 1.1.

#new-token: 3 and #cached-token: 2 add up to the five prompt tokens. This pair is the most useful line that SGLang prints. Section 3.4 uses it alone to find a prompt that misses the cache.

Fig. 1.3one /generate request through the three processessequence
One /generate request through the three processes Four lifelines: the client, the tokenizer manager, the scheduler, and the detokenizer. The client posts a five-token prompt. The tokenizer manager sends token ids to the scheduler over ZMQ. The scheduler matches two cached tokens in the radix cache, prefills three, and returns the first token after 68 ms. Eleven decode steps follow, and the detokenizer returns the text, for 725 ms end to end. client TokenizerManager Scheduler Detokenizer 1 POST /generate The capital of France is 2 5 token ids one ZMQ message 3 match_prefix 2 tokens cached 4 prefill 3 new tokens 5 first token 68 ms after the request 6 11 more tokens decode steps, 657 ms 7 text Paris. The capital of Italy is 8 200 OK e2e_latency 0.725 s
The tokenizer manager, the scheduler, and the detokenizer each touched the request once, and the prefill computed 3 of 5 tokens. Read top to bottom. Numbers are from our L4 request in section 1.1.

Our position

Choose SGLang for the shape of your traffic, never for a headline number. Agents, chat with a fixed system prompt, and retrieval with a fixed preamble all share long prefixes. For that traffic, the prefix cache saves prefill work on every request. If every prompt is unique and long, measure two engines on your own prompts before you decide.

Distrust any SGLang benchmark that does not name its version. The most repeated number on the web, "SGLang 29% faster than vLLM", comes from one offline test that lists SGLang v0.2.3. Section 5.3 shows where independent tests agree, and our own head-to-head runs.

To see how the scheduler, the cache, and each forward pass do this work, read Parts 2 to 4. Start with section 2.1, which follows the request through the three processes.

SGLang 0.5.22 · baa9bef · 2026-10-13 · Edition 2026.10