You have an NVIDIA GPU and a model name, and you want your own OpenAI-compatible endpoint. When you finish this section, you can install vLLM, start a server, send it a request, and read the startup log.

We use vLLM 0.31.0 in every command. vLLM publishes a new minor release about every two weeks, so pin the version in your install command.

Check the GPU first

First, check that the driver sees your GPU. Run this command:

nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv

You see one line for each GPU, with its name and its memory. If the command is not found, install the NVIDIA driver before you continue. vLLM does not install the driver for you.

Check the rest of your machine against this table. It comes from the vLLM 0.31.0 install guide. We did not test the lower limits.

Install vLLM

Pick one of three paths. Use the first path unless you have a reason to pick another.

Path 1, uv. uv creates a clean Python environment and picks the PyTorch build that matches your CUDA driver.

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm==0.31.0 --torch-backend=auto

Path 2, pip. Use a new virtual environment with Python 3.12, then install the same version.

python3.12 -m venv .venv && source .venv/bin/activate
pip install vllm==0.31.0

Path 3, Docker. The official image holds vLLM, PyTorch, and CUDA. We pulled vllm/vllm-openai:v0.31.0 on our arm64 DGX Spark, and it ran.

docker run --rm --gpus all --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:v0.31.0 Qwen/Qwen3-0.6B --reasoning-parser qwen3

The image starts the server for you, so with Docker you skip the next step. Mount your Hugging Face cache, or the container downloads the weights again on every start. Read the log with docker logs on the container.

Check the install with vllm --version. You see 0.31.0.

On the uv and pip paths, the first start compiles FlashInfer kernels, so it needs a CUDA toolkit with nvcc, and CUDA_HOME set to it. The Docker image has one. We installed vLLM 0.31.0 with uv on a host with only a driver, and every start stopped with Could not find nvcc. Install the toolkit, set CUDA_HOME, and start again. The troubleshooting page covers this and a toolkit that does not match.

Start the server

Now, start a small model. Qwen3-0.6B has 0.6 billion parameters, so its BF16 weights need about 1.2 GB.

vllm serve Qwen/Qwen3-0.6B --reasoning-parser qwen3 2>&1 | tee server.log

The server listens on port 8000. The first start downloads the weights, so it takes longer than the next start.

The tee command shows the log in the terminal and also writes it to server.log. Later pages search that file with grep, so keep this name for every start.

Qwen3 is a reasoning model. It writes its thinking first, between <think> tags, and then the answer. --reasoning-parser qwen3 tells vLLM to move the thinking out of the answer text. Without it, the thinking stays at the start of the answer. The reasoning models page explains the parser.

The startup log is long. Six lines tell you most of what you need. The listing shows them from our DGX Spark, where we served the larger Qwen3-30B-A3B. Your lines have the same shape with smaller numbers.

(APIServer pid=1) INFO 10-21 10:53:02 [api_utils.py:347]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.31.0
(EngineCore pid=143) INFO 10-21 10:53:14 [gpu_worker.py:479] Using V2 Model Runner
(EngineCore pid=143) INFO 10-21 10:53:17 [cuda.py:526] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=143) INFO 10-21 10:58:45 [model_runner.py:407] Model loading took 56.93 GiB memory and 330.542793 seconds
(EngineCore pid=143) INFO 10-21 10:59:46 [gpu_worker.py:692] Available KV cache memory: 31.32 GiB
(EngineCore pid=143) INFO 10-21 10:59:46 [kv_cache_utils.py:2464] GPU KV cache size: 234,960 tokens, Maximum concurrency for 8,192 tokens per request: 28.68x

Read the lines in this order:

1. The version. It must match the version you pinned.
2. The model runner. vLLM 0.31.0 uses the V2 model runner on CUDA by default. Some features switch it back to V1, and this line shows it.
3. The attention backend. vLLM picks one for your GPU. A change to the KV cache data type can change this line.
4. Model loading. This line gives the memory that the weights took and the time to load them.
5. The available KV cache memory. This line gives the memory that is left for the KV cache after the weights and the startup profile.
6. The KV cache size. This line gives the tokens that fit in the cache and the number of full-length requests that fit at the same time. Part 2 starts from this line.

The server is ready when the log shows Application startup complete.. On our DGX Spark, that took about eight minutes for 57 GiB of weights. The model loading line gives 330.5 seconds, and 325 of those seconds went to reading the weight files.

Send the first request

Open a second terminal. Send one chat request with curl:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3-0.6B", "messages": [{"role": "user", "content": "Name three prime numbers."}], "max_tokens": 64, "chat_template_kwargs": {"enable_thinking": false}}'

The chat_template_kwargs field turns the thinking off for this one request, so 64 tokens are enough for the answer. The reply is one JSON object. The answer text is in choices[0].message.content, and the token counts are in usage. In the server log, a new line shows the request path and 200 OK.

If you leave the thinking on, the model can spend all 64 tokens on it. The thinking then goes to the reasoning field, content is null, and finish_reason is length. We saw exactly this on an L4. Give a thinking request a much larger max_tokens.

The server speaks the OpenAI API, so the OpenAI Python client works without a change to your code:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
stream = client.chat.completions.create(
    model="Qwen/Qwen3-0.6B",
    messages=[{"role": "user", "content": "Name three prime numbers."}],
    stream=True,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

The tokens print one by one as the server streams them. That is your first streamed token.

When the output is different

• The server stops with an out-of-memory error. Another program uses the GPU, or the model is too large. Read the out-of-memory page.
• The answer starts with a long <think> block. The server runs without a reasoning parser. Start it with --reasoning-parser qwen3, or send "chat_template_kwargs": {"enable_thinking": false} in the request.
• The answer is empty and finish_reason is length. The thinking used all of max_tokens. Raise max_tokens, or turn the thinking off for the request.
• The request returns 404. The model name in the request must match the name in vllm serve. Run curl http://localhost:8000/v1/models to see it.
• The answers use different sampling than you set. By default, vLLM applies the sampling settings from the model repository. Start the server with --generation-config vllm to use the vLLM defaults.
• The start takes many minutes. The first start downloads the weights. On a DGX Spark, weight loading is slow in 0.31.0 for large models.

Your server started and you want to know that it works before you send it real traffic. When you finish this section, you can check the health route, the model list, the metrics, and the KV cache size.

Ask the health route

Run this command:

curl -i http://localhost:8000/health

You see HTTP/1.1 200 OK and an empty body. The server log shows the same request.

INFO:     Application startup complete.
INFO:     127.0.0.1:35152 - "GET /health HTTP/1.1" 200 OK
INFO:     127.0.0.1:34616 - "GET /v1/models HTTP/1.1" 200 OK

The health route answers 200 when the engine runs. It does not generate a token, so a 200 does not prove that a request completes. Send one short request as well, as in the first request.

If the route does not answer, the server is still loading or it stopped. Read the end of the server log. A load can take minutes for a large model.

Check the model name

Run this command:

curl http://localhost:8000/v1/models

The reply lists one model. Its id is the name your requests must use. Its max_model_len is the longest request in tokens that the server accepts.

Read the KV cache size

Go back to the startup log and find the line that starts with GPU KV cache size. On our DGX Spark it reads:

(EngineCore pid=143) INFO 10-21 10:59:46 [kv_cache_utils.py:2464] GPU KV cache size: 234,960 tokens, Maximum concurrency for 8,192 tokens per request: 28.68x

The first number is the total tokens of context that the server can hold for all users together. The second number is how many requests of the full maximum length fit at the same time. Shorter requests fit in larger numbers.

If the first number is much smaller than you expected, another program holds GPU memory, or the memory fraction is low. The memory page shows how to predict this number before you start.

Read the metrics

Run this command:

curl -s http://localhost:8000/metrics | grep -E "^vllm:(num_requests_running|num_requests_waiting|kv_cache_usage_perc)\{"

You see three lines. With no traffic, the running and waiting counts are 0 and the KV cache usage is 0. Send a request in a second terminal and run the command again to see the counts change.

Three metrics give the first answer to most questions:

A waiting count above 0 with the cache usage near 1 means the cache is full. The production page gives the full alert list.

You sent a request and the answer came back, slowly at first and then token by token. When you finish this section, you can name the two phases of a request and the four numbers that every benchmark in this manual reports.

Our result: on our DGX Spark, 64 users got eight times the output of one user. Each user waited about 7.5 times longer for every token.

Two phases

Prefill: the step that reads the whole prompt at once and writes its keys and values into the KV cache. One prefill step handles many prompt tokens, so the GPU is busy with math.

Decode: the steps that write the answer, one token per request per step. Each step reads all the model weights to make one token for each running request. The GPU waits for memory more than for math.

vLLM does not run the two phases as separate stages. Each scheduler step has one token budget. Running requests get their next decode token first. Then the scheduler spends the rest of the budget on prompt tokens, in chunks. A long prompt can take several steps. Part 8 explains why vLLM works this way.

Four numbers

One server, two loads

The numbers trade against each other as users grow. Here is our DGX Spark serving Qwen3-30B-A3B with vLLM 0.31.0, from one user to 64 users at the same time.

With 64 users, the server wrote about eight times as many tokens per second. Each user waited almost four times as long for the first token, and each token took more than seven times as long. A busy server is cheaper per token and slower for each user.

Every tuning page in this manual moves one of these four numbers. Each page says which number it moves and which number it costs.

Your first server works, and now other programs must call it. When you finish this section, you can set the address, the model name, the API key, the chat template, and the logging.

We did not measure the speed of these options. We checked the address, the model name, the API key, the YAML file, and the removed log flag on an L4 with the small Qwen3-0.6B. The rest comes from the vLLM documentation and the source at 0.31.0.

Pick the address and the port

The default listens on every interface, so other machines on the network can reach the server. For a test on your own machine, set --host 127.0.0.1.

Give the model a stable name

By default, the model name in the API is the name you passed to vllm serve, such as Qwen/Qwen3-8B. Clients must send that exact name. --served-model-name sets the name that clients use instead:

vllm serve Qwen/Qwen3-8B --served-model-name chat

Clients now send "model": "chat". You can then move to a new checkpoint without a change in the clients. The name also labels the metrics as model_name. If you give more than one name, the server answers to each of them.

Require an API key

--api-key makes the server require the key in the Authorization header. You can also set it with the VLLM_API_KEY environment variable, which keeps the key out of the process list.

Stop any server that already runs on port 8000. Then start the server with a key in the first terminal:

VLLM_API_KEY=change-me vllm serve Qwen/Qwen3-8B --host 127.0.0.1 2>&1 | tee server.log

When the server is ready, send a request with the key from a second terminal:

curl http://localhost:8000/v1/models -H "Authorization: Bearer change-me"

The key protects only the routes under /v1, /v2, /inference, and /cohere. Other routes on the same port, such as /invocations, /metrics, and /health, stay open. We checked this on an L4. Without the key, /v1/models returned 401. /health and /metrics returned 200, and so did a chat request to /invocations.

So the key does not make the server safe on a public network. Put a gateway in front of it, as the production page says.

Fix the chat template

vLLM formats chat messages with the chat template from the model repository. Some repositories have no template or a wrong one. Then the model sees a badly formatted prompt and answers poorly.

--chat-template takes a file path or the template text. --default-chat-template-kwargs sets template values for every request, such as '{"enable_thinking": false}' for Qwen3. A request can still set its own chat_template_kwargs. The reasoning models page uses this for Qwen3.

Control what the log keeps

With --enable-log-requests at the DEBUG level, the request lines include the prompt. Do not use that setting on a server that handles private data.

Older releases had a --disable-log-requests flag. At 0.31.0 that flag does not exist, and the request lines are off unless you turn them on.

Turn off usage statistics

vLLM sends anonymous usage data about the hardware and the model configuration by default. To turn it off, set VLLM_NO_USAGE_STATS=1 or DO_NOT_TRACK=1 in the environment of the server.

Keep the options in a file

vllm serve --config config.yaml reads the options from a YAML file. Each key is the long name of a flag without the two dashes, such as served-model-name: chat. An option on the command line wins over the same option in the file.

Check the options

Read the non-default args line near the top of the log. It lists every option that you changed from its default:

grep "non-default args" server.log

Then call /v1/models with and without the key. Without it, the server returns HTTP 401.

You chose a GPU and the server holds fewer users than you planned. When you finish this section, you can predict the KV pool and the number of users for a model and a context length.

Our result: on our L4 with Qwen3-8B, the 24,096-token pool fit 37 users with 640 tokens each, although the startup line printed 2.94.

Count the bytes for one token

Every token in a running request keeps its keys and values in GPU memory until the request ends. Four numbers in the model config give the size:

KV bytes per token = layers x KV heads x head size x 2 x bytes per value

The factor 2 counts the key and the value. Read the first three numbers from config.json as num_hidden_layers, num_key_value_heads, and head_dim. In BF16, a value takes 2 bytes. With an FP8 KV cache, it takes 1 byte.

The mixture-of-experts model has almost four times the weights of the dense model, but it keeps fewer bytes for each token. The KV heads decide this number, not the parameter count.

Turn bytes into users

vLLM gives the KV cache what is left after the weights and the startup profile. Divide that memory by the bytes per token to get the pool in tokens. Then divide the pool by the context that one user holds:

users that fit = KV pool in tokens / (prompt tokens + output tokens per user)

A request that needs more blocks than are free waits in the queue. If a running request needs a new block and none is free, vLLM stops it and computes it again later.

Check the count against the startup log

Read two lines after every start. On our DGX Spark, with Qwen/Qwen3-30B-A3B-Instruct-2507 and --gpu-memory-utilization 0.75, the log said:

(EngineCore pid=143) INFO 10-21 10:58:45 [model_runner.py:407] Model loading took 56.93 GiB memory and 330.542793 seconds
(EngineCore pid=143) INFO 10-21 10:59:46 [gpu_worker.py:692] Available KV cache memory: 31.32 GiB

At 96 KiB per token, 31.32 GiB holds about 342,000 tokens.

Fixed pool: in our sweeps, we set the pool size by hand with --num-gpu-blocks-override, which sets the number of 16-token blocks. We gave vLLM the same pool as the SGLang server on the same machine, so that the two engines had the same room. On the DGX Spark the pool was 234,960 tokens, and on the L4 it was 24,096 tokens. Your server sizes the pool from free memory, so it uses the full number.

The next line gives the pool in tokens and the users that fit at the full context length:

INFO 10-17 20:13:17 [kv_cache_utils.py:2464] GPU KV cache size: 24,096 tokens, Maximum concurrency for 8,192 tokens per request: 2.94x

The Maximum concurrency number assumes that every user fills the full --max-model-len. Real users hold less, so more of them fit. With 512 prompt tokens and 128 output tokens, the 24,096-token pool holds 37 users at their full length. During our load sweep, the log showed 40 running requests with the cache 99.5 percent full, because users early in their answer hold fewer tokens.

Decide what to change

If the count is too small for your traffic, you have four choices. They are in order of cost:

1. Lower --max-model-len to the longest context that your users send. The pool stays the same size, and one long request can no longer take all of it.
2. Use an FP8 KV cache. It halves the bytes per token. The FP8 KV page shows the cost.
3. Use FP8 weights. They free memory for the cache. The FP8 weights page shows when this pays.
4. Use a GPU with more memory, or split the model across GPUs. The parallelism page covers this.

The server stopped before it printed the KV cache size line. When you finish this section, you can read the reason from the last lines of the log and pick the one flag that fixes it.

Know what vLLM claims at startup

vLLM does four things before it serves a request:

1. It loads the weights.
2. It runs one profile pass with a full batch to measure the activation memory.
3. It captures CUDA graphs.
4. It gives the KV cache the rest of its share of the GPU.

The share is --gpu-memory-utilization. The default is 0.92, so vLLM claims 92 percent of the total GPU memory for itself. The number is a limit for one vLLM instance. It is not a fraction of the memory that is free when the server starts. On unified memory, such as a DGX Spark, it is not always a hard limit, as the DGX Spark page explains.

Find your failure in the log

Read the last 30 lines of the server log. Most startup failures match one of three cases.

Fix a shared GPU

On a shared GPU, set the fraction yourself. We ran vLLM 0.31.0 on an L4 that another team also used. Their processes held part of the memory, so we set --gpu-memory-utilization 0.83 and a pool of 2,928 tokens.

The first start still failed, because we also set the maximum length to 2,928 tokens. One request at that length needed slightly more KV cache than was free, and the error said that 2,912 tokens would fit:

(APIServer pid=3681321) INFO 10-22 00:35:04 [api_utils.py:286] non-default args: {'model_tag': 'Qwen/Qwen3-8B', 'host': '127.0.0.1', 'port': 30100, 'model': 'Qwen/Qwen3-8B', 'max_model_len': 2928, 'gpu_memory_utilization': 0.83, 'num_gpu_blocks_override': 183}
(EngineCore pid=3682071) ERROR 10-22 00:36:18 [core.py:1433] ValueError: To serve at least one request with the model's max seq len (2928), (0.4 GiB KV cache is needed, which is larger than the available KV cache memory (0.4 GiB). Based on the available memory, the estimated maximum model length is 2912. Try increasing `gpu_memory_utilization` (which also controls CPU memory on the CPU backend) or decreasing `max_mod

We set --max-model-len 2048, which is enough for our 640-token requests, and the server started. Later we raised it to 2,304 tokens, so that our tests with 2,048-token prefixes fit.

So on a shared GPU, the fraction and the maximum length work together. Lower the maximum length to the longest request you send before you lower the fraction.

Set the KV cache size directly

If you know the KV cache size you want, set it in bytes with --kv-cache-memory-bytes. vLLM then ignores --gpu-memory-utilization for the cache. Use this when two vLLM instances share one GPU and each one needs a known pool.

--num-gpu-blocks-override sets the number of 16-token blocks for a test, as in our fixed pool. Do not use it in production, because it does not check that the memory exists.

Check the fix

Start the server again with tee server.log, as in the first start, and wait for the pool line:

grep "GPU KV cache size" server.log

If the line is there, the server has a pool. Compare its size with your count.

You run vLLM on a DGX Spark or another machine where the CPU and the GPU share memory. When you finish this section, you can pick a memory fraction that leaves room for the operating system. You also know why the first start takes minutes.

Our result: with a fraction of 0.75, the weights took 56.93 GiB and the KV cache 31.32 GiB of the 121.69 GiB of memory.

Know what "GPU memory" means here

The GB10 in a DGX Spark has no separate GPU memory. The CPU and the GPU use the same LPDDR5x memory, which NVIDIA sells as 128 GB. On our machine, the startup log reported 121.69 GiB in total. The operating system, the page cache, and every other program use the same memory.

vLLM 0.31.0 knows this. On an integrated GPU it reads free memory from the operating system. The GPU driver reports too little free memory on these machines. It also releases its cached GPU memory when the whole system is more than 80 percent full. So --gpu-memory-utilization is a share of the whole machine, not of a separate card.

Do not trust the fraction as a hard limit

The fraction sets the share that vLLM plans for. On unified memory it does not always limit what vLLM takes. Reports from other users show two cases. We did not see them in our run:

• Issue 46307 reports a GB10 at a fraction of 0.70. It used about 112 GiB during the startup profile, against a share of about 83 GiB. The machine stopped responding. The issue was closed as stale with no fix.
• Issue 44740 reports a negative estimate of the CUDA graph memory with MTP speculation on the GB10. The KV cache grew too large, and the start ran out of memory. The issue is open.

In our run, the numbers matched the share of 0.75. That is one run, with no speculation.

To give the cache a fixed size, set it in bytes with --kv-cache-memory-bytes, as the out-of-memory page explains. Check that the weights, the cache, and a few GiB for the profile leave room for the operating system.

Pick the fraction

Add the weights, the KV cache you want, and a few GiB for the startup profile and the CUDA graphs. Divide the sum by the total memory. Leave room for the operating system and the page cache, which use the same memory.

Our run with Qwen/Qwen3-30B-A3B-Instruct-2507 in BF16 used a fraction of 0.75:

At 96 KiB per token, the 31.32 GiB holds about 342,000 tokens. That is enough for more than 500 users with 640 tokens each. Our sweeps used a smaller fixed pool. The fraction left about 30 GiB for the operating system and every other program.

We ran one fraction only. A sweep of the fraction against the KV pool and the swap activity is planned and not run yet.

Expect a slow first start

The first start took more than five minutes. The log shows where the time went:

(EngineCore pid=143) INFO 10-21 10:58:45 [model_runner.py:407] Model loading took 56.93 GiB memory and 330.542793 seconds

The load copies each tensor from the memory-mapped safetensors file into the GPU allocation, one tensor at a time. On the GB10, a copy from a file-backed page is slow. An open vLLM issue measured about 1.7 ms for each tensor copied from the file view. A copy in host memory first took about 0.2 ms.

Release 0.31.0 made a different part of the load faster. An indexed lookup of expert names saves about 25 s on a DGX Spark. The cost of each tensor copy stays.

Two things help:

• Keep the weights on the local NVMe drive, not on a network mount.
• Start the server once and keep it running. A restart pays the full load again.

Check the fraction

After the start, read the two memory lines and the system memory:

grep -E "Model loading took|Available KV cache memory|GPU KV cache size" server.log
free -g

If free -g shows almost no available memory while the server is idle, lower the fraction by 0.05 and start again. If the KV cache line is larger than your share allows, set the cache size in bytes instead.

Your server queues requests because the KV cache is full. When you finish this section, you can double the KV pool and check the cost in speed and accuracy.

Our result: on our L4, the first FP8 start failed, and the Triton backend fixed it. With the pool held fixed, one user got 2.3 percent faster per token.

Turn it on

The KV cache stores two values for each layer, head, and token. By default, vLLM stores them in the model data type, BF16 for Qwen/Qwen3-8B, at 2 bytes per value. FP8 stores each value in 1 byte:

vllm serve Qwen/Qwen3-8B --kv-cache-dtype fp8

The weights stay in BF16. Only the KV cache changes.

Count what it gives you

Use the formula from the memory page with 1 byte per value. For Qwen/Qwen3-8B, a token needs 144 KiB in BF16 and 72 KiB in FP8.

The same memory holds twice the tokens. So twice the users fit at the same length, or each user can have a context twice as long.

Fix the start on an L4

On our L4, the first start with --kv-cache-dtype fp8 failed. The cause is the attention backend:

1. vLLM tries FlashAttention first on the L4.
2. FlashAttention in vLLM 0.31.0 reads an FP8 KV cache only on Hopper and Blackwell data center GPUs. The L4 is an Ada GPU, SM 8.9.
3. vLLM moves to FlashInfer, the next backend in its list.
4. FlashInfer builds its kernels on the first start. On our node, that build failed.

(EngineCore pid=814129) INFO 10-22 09:11:03 [cuda.py:526] Using FLASHINFER attention backend out of potential backends: ['FLASHINFER', 'TRITON_ATTN'].
(EngineCore pid=814129) ERROR 10-22 09:11:53 [core.py:1433] /data/vllm/.venv/lib/python3.12/site-packages/flashinfer/data/cccl/libcudacxx/include/cuda/std/__cccl/cuda_toolkit.h:41:8: error: #error "CUDA compiler and CUDA toolkit headers are incompatible, please check your include paths"
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

The FlashInfer sampler fails to build in the same way on this node. So our sweep tool starts vLLM with VLLM_USE_FLASHINFER_SAMPLER=0.

Pick the Triton backend. It reads an FP8 KV cache and does not use the FlashInfer build. Also turn off the FlashInfer sampler, as our sweep tool does:

VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve Qwen/Qwen3-8B --kv-cache-dtype fp8 --attention-backend TRITON_ATTN

The attention backend page explains how vLLM picks a backend.

See what our run shows

We measured the FP8 KV cache with the Triton backend against the BF16 default on our L4 with Qwen/Qwen3-8B on 2026-10-22. Both ran in the throttled session, at 1 and 32 users. Raw runs

We held the pool at 24,096 tokens in both runs, as a fixed pool. So this run does not show the larger pool. It shows only the change in speed.

At one user, FP8 was 2.3 percent faster per token, at the same clock, and the seed ranges did not overlap. At 32 users, the seed ranges overlapped, and the FP8 run had a higher clock. So we do not claim a gain at 32 users.

Two things changed together: the KV data type and the backend. The table does not show which one gave the gain. The GPU was throttled for both runs, so read the ratio, not the times.

Check the accuracy

On 2026-10-23 we ran the first 500 GSM8K questions, 8-shot and greedy, on our L4. Qwen3-8B has no KV scales, so the FP8 KV runs used a scale of 1.0. The FP8 KV runs used the Triton backend, as above.

engine   config                     correct   accuracy   Wilson 95% interval
…
vLLM     BF16                       459/500   91.8%      89.1 to 93.9
vLLM     FP8 weights                453/500   90.6%      87.7 to 92.9
vLLM     FP8 KV cache               454/500   90.8%      87.9 to 93.0
vLLM     FP8 weights + FP8 KV       453/500   90.6%      87.7 to 92.9

Every interval overlaps the BF16 interval, so this run did not show a difference. With 500 questions, it cannot rule out a loss of a few points. That is one model and one benchmark, so run the same check on your model and your task. A model with a different attention layout can lose more.

The post "The State of FP8 KV-Cache and Attention Quantization in vLLM" on the vLLM blog, of April 2026, tested uncalibrated FP8 KV caches. Its authors are from AWS and Red Hat. In their tests, most models stayed close to BF16. Some did not. They name Kimi-K2.5, which uses the FlashMLA backend, as a model that did not work well with uncalibrated scales. We did not measure these results ourselves.

Check that your scales load

A checkpoint with calibrated scales does not always use them. At 0.31.0, issue 54623 reports that the Qwen3.5 model family never loads its KV scales. Issue 60333 reports the same for one MTP draft loader. Both issues are open.

First, look for the scales in the checkpoint. Search the weight index in the Hugging Face cache for tensor names that end in k_scale or v_scale. Change the model folder name to your model:

grep -oE '"[^"]*(k|v)_scale"' ~/.cache/huggingface/hub/models--Qwen--Qwen3-8B/snapshots/*/model.safetensors.index.json | head

No output means that the checkpoint has no scales.

A checkpoint with one weight file has no index. For it, list the tensor names with the safetensors Python package.

Then read the startup log. The warning Using KV cache scaling factor 1.0 means that vLLM uses a scale of 1.0. If the checkpoint has scales and the log still shows the warning, the loader dropped them.

Last, run your own prompts with and without FP8, and compare the answers before you go to production.

Know what changed after 0.31.0

Release 0.32.0 adds an NVFP4 KV cache on SM 8.x and SM 12.x GPUs, which include the L4 and the GB10. It stores fewer bytes per token than FP8. We did not test it.

Your model fits, but only with a small KV pool. When you finish this section, you can decide between a BF16 and an FP8 checkpoint at the same memory budget.

Our result: on our clock-locked L4, the FP8 checkpoint gave 36 percent more output than BF16 at 32 users, at the same pool.

Pick a checkpoint

An FP8 checkpoint stores most weights in 1 byte instead of 2. Many model makers publish one next to the BF16 model. For Qwen/Qwen3-8B, the FP8 checkpoint is Qwen/Qwen3-8B-FP8, and you serve it like any other model:

vllm serve Qwen/Qwen3-8B-FP8

vLLM reads the quantization from the checkpoint config, so you set no extra flag. If no FP8 checkpoint exists, vLLM can quantize a BF16 model when it loads, with --quantization fp8_per_tensor. We did not test that path.

Know what your GPU runs

The L4 is an Ada GPU, SM 8.9, and it has FP8 math. So vLLM runs the matrix math of the model in FP8, and the speed can rise as well as the free memory.

GPUs before Ada, such as the A100, have no FP8 math. On those, vLLM keeps the weights in FP8 and runs a weight-only kernel. You keep the memory saving, but expect less speed gain. We did not test such a GPU.

Count the memory it frees

The weights take close to half the memory in FP8. vLLM gives the memory that it frees to the KV pool. For Qwen/Qwen3-8B, each GiB of KV pool holds about 7,300 tokens in BF16, as the FP8 KV cache page shows.

In our runs, we held the pool at 24,096 tokens as a fixed pool, so that both runs had the same room. So we did not measure the larger pool. The gain in room comes from the bytes, not from a run.

See what our runs show

We measured Qwen/Qwen3-8B-FP8 against Qwen/Qwen3-8B on our L4 on 2026-10-22, in the session locked at 900 MHz. We ran 1 and 32 users with two seeds each. Raw runs

At 32 users, FP8 gave 36 percent more output and cut the time per token by 30 percent. The seed ranges did not overlap for either number. The BF16 clock was about 2 percent lower, much less than the gap.

At one user, the BF16 run fell to 210 MHz, below the lock, because the GPU throttled. So we do not compare the two at one user.

Use a 4-bit or 8-bit integer checkpoint

FP8 halves the weights. A 4-bit checkpoint stores most weights in about half a byte, so a model with twice the parameters can fit. On a 24 GB GPU such as the L4, a 4-bit checkpoint is a common way to serve a 14B model.

vLLM 0.31.0 reads these formats from the checkpoint config, so you set no flag for a published checkpoint:

For example, Qwen publishes a 4-bit AWQ checkpoint of Qwen3-14B:

vllm serve Qwen/Qwen3-14B-AWQ --max-model-len 8192

We started this AWQ checkpoint once on an L4, with no --quantization flag. The log showed quantization=auto_awq and the Marlin kernel. The weights took 9.44 GiB, and the pool held 36,272 tokens. We did not measure its speed or its accuracy.

We also started two of the other formats on the shared second GPU of our L4 node on 2026-10-23. That GPU was throttled, so we give no speeds.

• bitsandbytes. Without the plugin, --quantization bitsandbytes stops at start with Unknown quantization method: bitsandbytes. Install the plugin first, with uv pip install vllm-bnb-plugin, or pip install vllm-bnb-plugin on the pip path. With it, Qwen/Qwen3-1.7B loaded in 1.33 GiB against 3.22 GiB in BF16, which is 41 percent. The pool grew from 92,960 to 108,512 tokens.
• GGUF. With the plugin installed, vllm serve unsloth/Qwen3-0.6B-GGUF:Q4_K_M --tokenizer Qwen/Qwen3-0.6B loaded 0.55 GiB of weights, and a chat request returned HTTP 200.

The 41 percent is for one small model. It includes the embeddings and the output layer, which stay in 16-bit, so the share can differ on a larger model. We did not measure the accuracy of either format.

A 4-bit model can lose more accuracy than an FP8 model, and the loss depends on the model and the method. Test both formats on your own prompts.

Check the accuracy

On 2026-10-23 we ran the first 500 GSM8K questions, 8-shot and greedy, on our L4, with Qwen/Qwen3-8B against Qwen/Qwen3-8B-FP8:

engine   config                     correct   accuracy   Wilson 95% interval
…
vLLM     BF16                       459/500   91.8%      89.1 to 93.9
vLLM     FP8 weights                453/500   90.6%      87.7 to 92.9
vLLM     FP8 KV cache               454/500   90.8%      87.9 to 93.0
vLLM     FP8 weights + FP8 KV       453/500   90.6%      87.7 to 92.9

FP8 weights scored 1.2 points lower, and the two intervals overlap. So this run did not show a difference, and it cannot rule out a loss of a few points. It is one model and one benchmark. Run your own prompts on both checkpoints, and compare the answers before you go to production. We did not run this check on the 4-bit formats.

You want a context longer than the default, or you serve a model that mixes attention layers with sliding-window or state-space layers. When you finish this section, you can set the maximum length on purpose and read what the cache does for each kind of layer.

We did not run a long-context or a hybrid model for this manual. This page comes from the vLLM documentation, the source at 0.31.0, and the release notes.

Set the maximum length on purpose

If you do not set --max-model-len, vLLM takes the length from the model config. For Qwen/Qwen3-8B that is 40,960 tokens. At 144 KiB per token, one request at that length needs 5.6 GiB of KV cache. On a 24 GB GPU, the 16.4 GB of weights can leave room for less than one such request, and then the start fails.

Set --max-model-len to the longest prompt plus answer that your users send. The pool size stays the same. A shorter limit only stops one request from claiming the whole pool, and it lets the start check pass.

Check the length after an upgrade

Some models extend their context with a scaling method such as YaRN. In release 0.30.0, vLLM changed its YaRN handling to match the Hugging Face Transformers library. For some models, the length that vLLM derives from the config became shorter. The release notes give one model that went from 131,072 to 32,768 tokens.

After an upgrade, read the derived length in the startup log. If it changed, set --max-model-len yourself, or check that the model still supports the longer length.

Know how a hybrid model uses the cache

A hybrid model has more than one kind of layer. vLLM puts each kind in its own KV cache group. The table gives the rules for each group.

The group design lets one pool serve all the kinds. It also means that the prefix-cache hit rate of a hybrid model differs from the hit rate of a full-attention model with the same prompts.

Avoid the removed and renamed flags

Hybrid support changes fast. Three changes between 0.29 and 0.32 matter:

• The all mode for the Mamba cache was deprecated in 0.30, and it made vLLM use the older model runner. Release 0.32 removed it.
• Release 0.31 renamed --enable-mamba-fine-grained-prefix-cache to --enable-mamba-shared-prefix-checkpoint.
• After the removal of the all mode, an open issue reports that --mamba-block-size has no effect.

If a hybrid model starts slower or uses more memory after an upgrade, check these flags first. --disable-hybrid-kv-cache-manager turns the groups off, as a test of whether the groups cause the problem.

Check the setup

Start the server and read the pool line. It gives the maximum concurrency at your maximum length:

grep "GPU KV cache size" server.log

The pool line prints only when the start succeeds. If the start stops with the error about the KV cache and the maximum length, lower --max-model-len until the pool line prints. Then read its concurrency, and compare it with the users you plan to serve at full length.

Your chat service streams smoothly for a few users and has long gaps between tokens for many. When you finish this section, you can keep the stream steady as the number of users grows.

Our result: on our L4, about two in three requests met our latency target at 16 users, and none at 32. Capping the batch did not fix it.

Know what a chat user feels

A chat user feels two numbers. The first is the time to the first token (TTFT). The second is the gap between tokens while the answer streams (ITL). Throughput is what you pay for, but users do not see it.

Watch the P99 ITL, the worst gap in 100. One long pause in a stream feels worse than a slightly slower stream. Set a target for both numbers. Our target was a first token within 1 second and 100 ms or less per token.

See where the stream breaks

We measured users at once from 1 to 64 on our L4 with Qwen/Qwen3-8B on 2026-10-21, in the full-clock session. Each request sent 512 prompt tokens and asked for 128 tokens, with three seeds per point. Raw runs

Up to 16 users, each new user added output, and the stream stayed steady. But at 16 users, one request in three already got its first token too late. At 32 users, the output still rose, but the P99 ITL grew seven times and no request met the target. The server was busy, and the users waited.

Treat results below 32 users as affected by prefix-cache reuse

This sweep kept one server for all the user counts, and it did not reset the prefix cache between them. Each seed sent the same first prompts at every count, so many requests found their prompt in the cache and skipped the prefill. On 2026-10-23 we ran the sweep again on the same L4, with a reset before each run and the clock locked at 900 MHz. We compared it with an unflushed run of that same session:

median TTFT, flushed against unflushed
users   vLLM                                   SGLang
4       650 / 145 ms (4.5x)                    532 to 645 / 141 ms (3.8 to 4.6x)
8       951 to 978 / 264 to 274 ms (3.6x)      1,174 to 1,273 / 173 to 178 ms (6.6 to 7.3x)
16      1,550 to 1,587 / 294 to 295 ms (5.3x)  1,804 to 1,944 / 218 ms (8.3 to 8.9x)
32      1,982 to 2,084 / 2,012 to 3,091 ms     3,216 to 3,228 / 2,710 to 2,917 ms
…

With the reset, 50 percent of the requests met our target at 8 users, against 75 to 81 percent without it. At 16 users the share fell from about 65 percent to 25 percent. The P99 ITL at 16 users was 522 ms with the reset, against 82 to 84 ms in the unflushed run of the same session. From 32 users the queue dominated, and the reset changed little. So the stream breaks between 8 and 16 users on this card, not between 16 and 32. The table above stays as a record of the full-clock session.

Each step has a budget of 2,048 tokens on this GPU. When new prompts arrive, their chunks share the step with the running streams, so the running streams pause. The token budget page explains the order.

Know what a cap on sequences does

--max-num-seqs sets how many requests run at once. On the L4 the default is 256, and the token budget page gives the rule for other GPUs. We set it to 16 and ran the same sweep:

The cap made each stream a little smoother. But half the users waited in the queue for their first token, and output fell by 38 percent. The cap run had the higher clock, so the clock does not explain the loss. At 64 users, the cap run fell to 630 MHz, so we do not compare that row.

A cap does not make room for more users. It only chooses who waits.

Try a cap at 64 users

At 64 users, the P99 ITL of vLLM on our L4 was more than a second, against about 150 ms for SGLang. On 2026-10-23 we tried to close that gap with a cap. The clock was locked at 900 MHz, and each run had two seeds and a cache reset. --max-num-active-seqs is a second cap at 0.31.0, on the requests that run in one step.

config                        median SM clock (s1/s2)   median TTFT       P99 ITL               output tok/s
SGLang default                900 / 900                 19.4 s            143 to 153 ms         225 to 231
…
vLLM default, earlier run     900                       11.8 s            1,359 ms              220
…
--max-num-seqs 32             900 / 795                 17.2 to 20.6 s    1,163 to 1,399 ms     196 to 230
…
--max-num-active-seqs 48      900 / 900                 11.8 s            1,329 to 1,373 ms     215 to 218
…
--max-num-active-seqs 16      795 / 683                 44.8 to 48.8 s    1,133 to 1,139 ms     129 to 136

No cap that we tried brought the P99 ITL of vLLM near the SGLang value. At 16 or 32 running requests, the P99 ITL stayed above 1.1 s, and the first token waited 17 to 65 s. Many runs in this block ran below the lock, and the file lists them all. So on this card, a cap does not fix the long gaps at 64 users. Fewer users per replica does.

See what a small pool does

The KV pool limits the users too. We measured a pool of 6,560 tokens, with --num-gpu-blocks-override 410 and --max-model-len 4096, against our pool of 24,096 tokens. Both ran on our L4 on 2026-10-22 in the session locked at 900 MHz, at 32 users and two seeds each.

A request of 640 tokens needs 640 tokens of pool by its end. So only about 10 requests fit in 6,560 tokens at once. The other 22 waited for a running request to finish.

vLLM starts a new request only when its blocks fit in the pool. A running request that needs a new block can push another one out, which vLLM counts as a preemption. So most of the 32 requests waited in the queue before they started.

The small-pool run fell to about 250 MHz, below the lock. A clock 3.6 times lower can make a step up to 3.6 times slower. The first token came 10 times later, so the queue caused most of the wait. Do not compare the throughput or the time per token of these two runs.

Tune it

1. Set your targets for the first token and the time per token.
2. Run your own load at more and more users, and find the last count that meets both.
3. If the KV cache usage is near 100 percent and requests wait, the pool is too small. Add room with an FP8 KV cache, or FP8 weights. A shorter maximum length does not help here, because the pool stays the same size.
4. If the pool has room but the P99 ITL is too high, the GPU is full. Add a GPU or a replica. A cap on sequences moves the wait to the first token.

On our DGX Spark, vLLM and SGLang gave the same output at 32 and 64 users. But the P99 ITL of vLLM was about twice as high, 478 against 220 ms at 32 users. So also compare the P99 ITL when you choose an engine for chat.

Every request repeats the same long system prompt. When you finish this section, you can order a prompt so that the cache hits, and prove the hit from a metric.

Our result: on our L4, the cache kept the first token near 400 ms at every prefix length. Without it, the first token took 1,213 ms at 1,024 shared tokens.

See what the cache saves

Prefix caching is on by default in vLLM 0.31.0. We ran the same workload with it on and off on the L4 with Qwen/Qwen3-8B. Each run sent 4 groups of 16 prompts at 8 users. The prompts in a group shared a system prompt of the length in the table. Each prompt then added a 128-token question and asked for 64 tokens back.

With the cache on, the first token took about 400 ms at every prefix length. With the cache off, the first token grew with the shared part, because every request computed the whole prompt again.

Read the last two rows with care. Without the cache, the GPU throttled. The median clock fell to 1,035 MHz at 2,048 tokens and to 675 MHz at 4,096 tokens. The cleanest comparison is the 1,024-token row, where both runs held 1,410 MHz or more. There the cache cut the first token about 3 times and gave 50 percent more output. The runs with the cache on were on 2026-10-17 and 2026-10-18, and the runs with it off on 2026-10-21.

The server log reports the hit rate every few seconds:

INFO 10-18 00:43:16 [loggers.py:320] Engine 000: Avg prompt throughput: 231.0 tokens/s, Avg generation throughput: 93.6 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.2%, Prefix cache hit rate: 87.2%

Put the shared part first

The cache matches a prompt from its first token, one 16-token block at a time. It stops at the first block that differs. So a cache hit needs the same tokens at the start of the prompt, in the same order.

1. Put the parts that never change first: the system prompt, the tool definitions, and the fixed instructions.
2. Put the parts that change rarely next, such as the documents of one session.
3. Put the parts that change on every request last: the user question, the time, and the request ID.

A timestamp or an ID near the start of the prompt changes the first block, and every later block then misses. Check your chat template too. Some templates put the date into the system prompt.

vLLM caches only full blocks. A shared part of 20 tokens gives one cached block of 16, and the last 4 tokens are always computed.

Keep tenants apart

Two users with the same prompt prefix share cached blocks. That saves compute, and it also lets one user's timing reveal what another user sent. To keep tenants apart, send a cache_salt field with each request. Requests with different salts never share blocks. The LoRA adapter is part of the block key too, so two adapters never share blocks.

Prove the hit

Read two counters from /metrics:

curl -s localhost:8000/metrics | grep -E "^vllm:prefix_cache_(queries|hits)"

The hits divided by the queries is your hit rate. For one prompt, send the same long prompt twice and compare the second time to first token with the first.

If the hit rate is low, the cause is almost always the prompt order. A full pool is the next cause, because the cache keeps old blocks only until a running request needs the memory.

Long prompts arrive and short requests stall behind them. When you finish this section, you can choose a batch token budget for your latency target.

Our result: on our DGX Spark at low load, a budget of 16,384 tokens raised the P99 ITL from 574 to 798 ms.

Know the budget

vLLM always splits long prompts into chunks. In each step, the running streams get their next token first, and a waiting prompt fills the rest of the budget. --max-num-batched-tokens sets the budget. The token budget page explains the order.

A larger budget lets a long prompt finish its prefill in fewer steps, so its first token comes sooner. But each of those steps takes longer, and every running stream waits for it. So at low load, a larger budget can give a faster first token and longer gaps between tokens.

Know your default

vLLM sets the budget and the cap on running requests from the GPU memory that it reads. The defaults also differ between vllm serve and the offline LLM class. Other manual pages link to this table.

Our DGX Spark has 121.69 GiB of memory that the CPU and the GPU share. Its startup log still shows a budget of 2,048, the value for a small or unknown GPU. Read the value that your server uses before you change it. The 0.31.0 log does not print the budget under its own name. Find compile_ranges_endpoints in the engine config line. The largest number in that list is the budget.

See what our runs show

We measured budgets of 2,048 and 16,384 tokens on our DGX Spark with Qwen3-30B-A3B on 2026-10-21. The prompts had 1,024 to 4,096 tokens, and each request asked for 64 to 256 tokens. We sent requests at a fixed rate, with three seeds per point. Raw runs

At 0.5 requests per second, the larger budget raised the P99 ITL by 39 percent, and the seed ranges did not overlap. The first token came sooner, but the seed ranges of the first token overlapped. So the cost is clear, and the gain is not confirmed.

The median first token at 0.5 requests per second was slower than at 1 request per second, for both budgets. With the 2,048 budget, all three seeds at 0.5 were above 900 ms. All three at 1 were below 600 ms. So one slow seed does not explain it. We did not find the cause.

At 1 and 2 requests per second, the two budgets gave the same numbers within the seed ranges. At those rates, more requests share each step, so a single long prompt fills less of the budget.

Read our result as narrow: two budgets, three low rates, one model, and one machine. The 2026 paper "P-PAS: Prefill-Pressure Adaptive Scheduling for Long-Context LLM Serving" found that no single fixed budget in vLLM was best at every load. A large budget gave lower latency at low pressure. A small budget won when several prefills competed with running streams. We did not measure that crossover ourselves.

The run with no budget flag matched the run with 2,048 at every rate. That agrees with the 2,048 in the startup log. All runs held a median clock of 2,398 to 2,411 MHz.

Tune it

1. Read the budget that your server uses.
2. If the P99 ITL is too high and long prompts arrive, lower the budget. Try 2,048 if your default is larger.
3. If the first token of long prompts is too slow and the P99 ITL has room, raise it.
4. Measure at your real request rate, because the effect changed with the rate in our runs.
5. Measure again when your load changes, because the best budget moves with the load.

vLLM also has --long-prefill-token-threshold. It caps the tokens that one long prompt gets in a step when other requests wait. It is off by default, and we did not test it.

A long conversation slows down as it grows. When you finish this section, you can keep earlier turns cached without a full prefill.

Why a long session slows down

Each turn of a chat sends the whole conversation again. Without a cache, the server computes every earlier turn on every request, so the first token takes longer with each turn.

The prefix cache removes that cost while the earlier turns stay in the cache. Turn 10 starts with the same tokens as turn 9, so the blocks of turn 9 are a hit. Only the new user message and the answer need compute.

The cache keeps a block only until a running request needs the memory. On a busy server, the blocks of a user who paused for a minute can be gone when that user sends the next turn. The whole conversation is then computed again.

Keep the prompt stable across turns

The hit needs the same tokens in the same order:

• Send earlier turns exactly as before. Do not trim, summarize, or reformat them between turns.
• Keep the system prompt and the tool list the same for the whole session.
• If you must shorten a conversation, cut whole turns from the middle, not from the start. A change at the start makes every later block miss.

Offload blocks to host memory

When the GPU pool is too small to keep idle sessions, vLLM can copy blocks to CPU memory before it drops them. A later request reloads them from CPU memory, which is faster than a full prefill of a long context.

The native offload needs one flag, the CPU memory to use in GiB:

vllm serve Qwen/Qwen3-8B --kv-offloading-size 32

--kv-offloading-backend picks the implementation. native is the default and uses the code in vLLM. lmcache uses the LMCache library, and it needs a separate LMCache server process. The offload trades CPU memory and PCIe transfer time for compute.

Not measured on our hardware. We have not timed a reload from CPU memory against a prefill on the L4 or the GB10.

Use a shared cache across servers

With several servers behind a router, a user's next turn can go to a server that never saw the earlier turns. Two fixes exist. A router that sends a session to the same server keeps the hit on that server. A shared KV store, through a KV connector such as LMCache or Mooncake, lets any server load the blocks. The KV connector page explains the second.

Check the session cache

Send turn 1 of a conversation, then turn 2 with turn 1 at its start. Read the prefix cache counters before and after turn 2:

curl -s localhost:8000/metrics | grep -E "^vllm:(external_)?prefix_cache_(queries|hits)_total"

During turn 2, vllm:prefix_cache_hits_total rises by about the length of turn 1, rounded down to full 16-token blocks. We checked this on an L4 with Qwen3-0.6B. Turn 1 had 2,182 prompt tokens. In turn 2 the counter rose by 2,176, which is 136 full blocks. With offload on, the startup log names the connector, OffloadingConnector for the native backend. A block that came back from CPU memory counts in the vllm:external_prefix_cache_hits_total counter.

Your application needs JSON that always parses. When you finish this section, you can request structured output and check its cost.

Ask for a schema

With structured output, vLLM limits each next token to the tokens that keep the output valid for your schema. The model cannot produce text that breaks the schema. You ask for it in the request, not with a server flag.

With the OpenAI client, use response_format:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
schema = {
    "type": "object",
    "properties": {"city": {"type": "string"}, "days": {"type": "integer"}},
    "required": ["city", "days"],
}
reply = client.chat.completions.create(
    model="Qwen/Qwen3-8B",
    messages=[{"role": "user", "content": "Plan a trip to Lisbon. Answer in JSON."}],
    response_format={"type": "json_schema", "json_schema": {"name": "trip", "schema": schema}},
)
print(reply.choices[0].message.content)

For other limits, send structured_outputs in extra_body. It takes one of json, json_object, regex, choice, grammar, or structural_tag. Release 0.12.0 removed the older guided_json and guided_* fields. A recipe that uses them is out of date.

Know what runs on the server

The default backend setting is auto, which picks a backend for the request. You can also name one: xgrammar, guidance, outlines, or lm-format-enforcer. The server compiles your schema into a grammar the first time it sees the schema, and it caches the result. On each step it builds a mask of the allowed tokens and applies it before sampling.

Two costs follow:

• The first request with a new schema waits for the compile.
• Each step builds a mask, which adds CPU time for each token.

Not measured on our hardware. We have not timed the compile or the mask cost on the L4 or the GB10.

Return tool calls

For tool calls, start the server with a parser for your model's tool format:

vllm serve Qwen/Qwen3-8B --enable-auto-tool-choice --tool-call-parser hermes

The parser turns the model's text into the tool_calls field of the response. Pick the parser that matches your model family. The vLLM documentation lists them.

Avoid two known problems

• Speculative decoding. A schema and a speculation method can disagree on the next tokens. An issue on 0.31 showed invalid JSON with MTP speculation for one model, and the issue is now closed. Test your schema with and without --speculative-config before you rely on both.
• Reasoning models. A model that thinks before it answers needs the schema to apply to the answer only. Set the reasoning parser for your model, so the limit starts after the thinking ends. The reasoning models page shows the flag.

Check the result

Parse every response in your client, even with structured output on. Count the parse failures and the requests that hit max_tokens before the JSON closed. A response cut by the token limit is not valid JSON.

You must process a large set of prompts overnight. When you finish this section, you can get the most tokens per second when latency does not matter.

Our result: on our L4 with Qwen3-8B, output stopped growing at 32 users, near 274 tokens per second.

Pick the offline class or the server

For a fixed list of prompts, the offline LLM class runs the engine in your Python process. It has no HTTP layer and no client to tune:

from vllm import LLM, SamplingParams

prompts = ["Summarize the history of Lisbon in three sentences.", "Write a haiku about the sea."]
llm = LLM(model="Qwen/Qwen3-8B")
params = SamplingParams(temperature=0.7, max_tokens=256)
outputs = llm.generate(prompts, params)
for out in outputs:
    print(out.outputs[0].text)

Pass the whole list in one call. The engine schedules all the prompts itself and keeps the batch full.

For a file of OpenAI requests, use the batch command:

vllm run-batch -i requests.jsonl -o results.jsonl --model Qwen/Qwen3-8B

Use the server when other programs must send the work, or when the job shares the GPU with live traffic.

Find where throughput stops growing

When nobody waits for a first token, only the total output matters. Raise the concurrency until the output stops growing.

We ran Qwen/Qwen3-8B with 512-token prompts on our L4, in the full-clock session on 2026-10-17. The output grew from 13.5 tok/s with one user to 273.9 tok/s with 32 users. At 64 users it fell to 241.4 tok/s, because the pool was full and the GPU throttled. In a separate run on 2026-10-18, a fixed rate of 4 requests each second gave 286.6 tok/s, at 1,245 against 1,275 MHz. So on this card, the most output came at about 32 users at once. More users only made each request wait longer.

The ceiling on your setup depends on the model, the prompt and answer lengths, and the pool. Run the sweep with your own lengths to find it.

Raise the limits for a batch job

Two limits cap the batch. Their defaults depend on the GPU memory and on how you run vLLM:

• --max-num-seqs caps the requests that run at once.
• --max-num-batched-tokens caps the tokens in one step. The offline LLM class has a larger default than the server. The vLLM documentation suggests a value above 8,192 for throughput, most of all for small models on large GPUs.

The token budget page gives the default of each limit for your GPU.

A larger token budget lets long prompts finish their prefill in fewer steps. Running streams get less of each step, which a batch job does not care about.

We measured the budget on our DGX Spark, where 16,384 gave longer gaps between tokens at a low rate. The long prompts page has the numbers. We did not measure it for a batch job.

Check the run

Watch vllm:num_requests_running and vllm:kv_cache_usage_perc. If the cache usage stays near 1 and requests wait, the pool limits the batch. Add memory with an FP8 KV cache or a shorter maximum length. If the running count sits at the --max-num-seqs limit with free cache, raise the limit.

You have many fine-tunes of one base model. When you finish this section, you can serve LoRA adapters per request from one base model.

Start the server with LoRA on

A LoRA adapter is a small set of extra weights on top of a base model. The server keeps one copy of the base model and applies the adapter that each request names. Requests for different adapters can run in the same batch.

vllm serve Qwen/Qwen3-8B \
  --enable-lora \
  --lora-modules support=/adapters/support legal=/adapters/legal \
  --max-loras 4 --max-lora-rank 64

Each path is a folder in the PEFT format that most training libraries write. It holds adapter_config.json and the weights in adapter_model.safetensors. vLLM also reads adapter_model.bin. A path can also be a Hugging Face repository name.

A request picks an adapter by its name in the model field:

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model": "support", "messages": [{"role": "user", "content": "Reset my password"}]}'

/v1/models lists the base model and each adapter.

We ran this setup on the shared second GPU of our L4 node on 2026-10-23. We used Qwen/Qwen3-0.6B and two public adapters of rank 8 and rank 64. /v1/models listed the base model, support, and legal, and a chat request to each adapter returned HTTP 200.

Size the three limits

The default of one adapter for each batch is low for a server with many adapters. Requests for a second adapter wait until the first adapter's batch is done. Raise --max-loras to the number of adapters that are busy at the same time.

Set --max-lora-rank to the largest rank of your adapters, not higher. The vLLM documentation advises against a large default such as 256, because the server reserves memory for the largest rank.

Load adapters while the server runs

To add an adapter without a restart, start the server with VLLM_ALLOW_RUNTIME_LORA_UPDATING=True. Then post to /v1/load_lora_adapter with the name and the path, and to /v1/unload_lora_adapter to remove it. Turn this on only when you trust every caller of the server, because the route loads files from a path that the caller gives.

Know what LoRA changes elsewhere

• Prefix cache. The adapter is part of each block key. Two adapters never share cached blocks, even for the same prompt.
• CUDA graphs. vLLM captures separate graphs for batches with an adapter. More adapters and more graph sizes add startup time and memory.
• Names. Since 0.31, an adapter cannot have the name of a served model.

Not measured on our hardware. We have not measured the cost of the adapter count or the rank on the L4 or the GB10.

You run evals and the scores change between runs. When you finish this section, you can get the same output for the same request.

Know why outputs change

The vLLM documentation says that vLLM does not promise the same output by default, for the sake of speed. Three things change the result of one request:

• The batch. The kernels split their sums by the batch size. The same request adds its numbers in a different order in a batch of 3 than in a batch of 30. Floating-point sums depend on the order. A small difference in one logit can change a sampled token, and every later token then differs.
• The schedule. In the default multiprocess server, the timing of requests decides which batch each one joins.
• The prefix cache. A request that reuses cached blocks computes its prompt in a different way than a request with no cache hit. In our runs on 0.31.0, a warm cache changed the greedy output of 11 of 32 prompts. The 2026 paper "Same Request, Different Answer" found the same effect on vLLM 0.11.
• The dtype. BF16 and FP16 round each number differently, so the same model in the two dtypes writes different text.

The seed defaults to 0 in the current engine, and you cannot unset it. Speculative decoding needs every worker to sample the same tokens. So a seed is always there, and it is not enough by itself.

Pick the fix for your case

Every fix also needs the same hardware, the same vLLM version, and the same settings. A new release or a new GPU changes the kernels, and the outputs change with them.

Turn on the batch-invariant mode

Set one environment variable when you start the server:

VLLM_BATCH_INVARIANT=1 vllm serve Qwen/Qwen3-8B 2>&1 | tee server.log

The mode needs an NVIDIA GPU with compute capability 8.0 or higher, which includes the L4 and the GB10. The vLLM documentation lists Qwen3-8B among the models it tested in this mode. The mode turns off some speed features that change the order of the sums, such as custom all-reduce with tensor parallelism.

We ran one probe with the small Qwen3-0.6B on an L4. With the mode on, a greedy prompt gave the same text alone and inside a batch of 32 requests. With the mode off, the two texts differed. One prompt on one small model proves little for yours, so run the test at the end of this page on your own model.

See what changed the output in our runs

We sent ShareGPT questions after one shared system prompt of about 170 tokens, greedy, with 128 output tokens and thinking off. We compared each output with the output of a second run, token by token. This was on the second GPU of our L4 node on 2026-10-22 and 2026-10-23, with Qwen/Qwen3-0.6B. That GPU is shared with its owner and had no clock lock.

A clock changes the speed of a run. It does not change the greedy tokens. Each count is still one session on one GPU.

comparison                     SGLang 0.6B   vLLM 0.6B
flushed repeat against fresh   0 of 32       0 of 32
cache off against cache off    0 of 32       0 of 32
fresh against warm cache       21 of 32 (45) 11 of 32 (57)
cache on against cache off     25 of 32 (26) 21 of 32 (48)

• One user, the same cache state. A repeat after a cache reset, or a repeat with the cache off, gave the same text for all 32 prompts.
• One user, a warm cache. The same prompts against a warm cache changed 11 of 32 outputs. The cache on against the cache off changed 21 of 32.
• Many users. At 64 users sent at once, two runs with the cache off differed in 50 of 64 outputs. The batch alone changed most outputs, so the cache effect could not be separated at that load.
• The dtype. We sent 64 questions at one user with the cache off. BF16 against FP16 changed 48 of 64 outputs for Qwen3-0.6B and 56 of 64 for Qwen3-1.7B. The first different token came at a median position of 51 and 56. A repeat in the same dtype changed none.

So at one user, a cache reset is enough to repeat a greedy output exactly. Under load, a reset was not enough in our runs. For a live server, the batch-invariant mode is the fix this page covers, and we did not test it at that load. Our counts come from small models and one GPU, so test your own model.

Know what the batch-invariant mode does not cover

At 0.31.0 the mode is in beta. Work by other teams shows three limits. We did not measure them ourselves:

• The speed cost. In the post "Defeating Nondeterminism in LLM Inference" of September 2025, Thinking Machines Lab ran 1,000 Qwen3-8B requests on vLLM. They took 26 s by default and 55 s with batch-invariant kernels, and 42 s with a better attention kernel. The 2026 CoRun paper reports that batch-invariant kernels more than double the latency and cut throughput by up to 74 percent.
• Other changes in execution. The 2026 paper "Vosti: Specifying, Implementing, and Verifying Deterministic LLM Inference" tested changes such as prompt chunking and KV cache reuse. The batch-invariant mode of vLLM gave different logits under some of them.
• Some model paths. At 0.31.0, a model that picks its own attention backend skips the check for the mode, so its output can still depend on the batch. Release 0.32.0 checks the backend of every attention layer. A fix for MoE models with data, tensor, and expert parallelism under torch.compile is not in 0.32.0.

So test the exact setup that you serve: the model, the parallel layout, the prefix cache setting, and the token budget.

Check the result

Run the same eval twice and compare the outputs token by token, not only the score. Between the two runs, reset the prefix cache or restart the server, as the benchmark page shows. If the outputs differ, send one failing prompt alone twice. If the two answers match when alone, the batch causes the difference.

Know the speed cost on our L4

We measured the cost on our L4 on 2026-10-23, with the clock locked at 900 MHz. The runs went in the order default, batch-invariant, default again, with two seeds each and a cache reset before each run.

…
users   median TPOT default / batch-invariant   output tok/s default / batch-invariant
1       6.9 / 16.9 ms (2.4x)                    140.6 / 58.7 (58% lower)
8       9.9 / 18.2 ms (1.8x)                    755 / 420 (44% lower)
32      18.8 / 25.9 ms (1.4x)                   1,565 / 1,154 (26% lower)
…

On Qwen/Qwen3-8B at 8 users, the time per token was 1.4 times longer and the output tok/s 27 percent lower. At 32 users they were 1.3 times longer and 23 percent lower. The 1-user runs of the 8B model ran below the lock, so we leave them out. The relative cost fell as more users shared each step, and it was larger on the small model. Measure it on your model before you turn the mode on for a live server.

Your RL trainer must generate rollouts and then update the weights. When you finish this section, you can name the control routes an RL framework calls.

The loop

An RL step has two halves. The rollout half generates answers with the current weights. The training half computes a loss and updates the weights. Then the generator needs the new weights before the next rollout.

On one set of GPUs, the two halves take turns on the same memory. The generator must give its memory back during training and take it again after. The vLLM server provides the pieces for this, and many RL frameworks call them. The vLLM documentation lists verl, OpenRLHF, TRL, NeMo-RL, SkyRL, Prime-RL, and others among its users.

The pieces

The routes below exist only when you start the server with VLLM_SERVER_DEV_MODE=1. They free memory, replace weights, and run methods on every worker. Never expose a server in this mode outside the network of the trainer.

Sleep mode. Start the server with --enable-sleep-mode. A call to /sleep frees GPU memory, and /wake_up takes it back. At the deeper of its two levels, sleep drops the weights too, which suits a step where the trainer sends new weights anyway. --sleep-preserve-parameter-names keeps frozen weights on the GPU, so they do not move each step.

Pause and resume. /pause stops new generation, and /resume starts it again. A pause can abort the running requests, wait for them, or keep them.

Weight updates. The trainer starts a transfer with /init_weight_transfer_engine, then calls /start_weight_update, sends the weights, and calls /finish_weight_update. The transfer can run over NCCL between GPUs or through CUDA memory handles on one machine. /update_weight_version records which version the next rollouts use.

Collective calls. /collective_rpc runs a method on every worker. Frameworks use it for setup that vLLM does not cover.

External launch. With --distributed-executor-backend external_launcher, a launcher such as torchrun starts the vLLM workers inside the trainer's processes. Some colocated frameworks use this path.

Try sleep and wake by hand

Start a server in development mode with sleep mode on. --host 127.0.0.1 keeps the control routes on your own machine:

VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen3-8B --enable-sleep-mode --host 127.0.0.1 2>&1 | tee server.log

In a second terminal, put the server to sleep, check it, and wake it:

curl -X POST 'http://localhost:8000/sleep?level=1'
curl http://localhost:8000/is_sleeping
curl -X POST 'http://localhost:8000/wake_up'
curl http://localhost:8000/is_sleeping

The two POST routes return HTTP 200 with an empty body. The first /is_sleeping call returns {"is_sleeping":true}, and the second returns {"is_sleeping":false}. Run nvidia-smi between the calls to see the GPU memory fall during the sleep and rise after the wake.

We ran these calls at level 1 with the small Qwen3-0.6B on an L4. Each POST returned 200 with an empty body, and the sleep took 1.9 s. The GPU memory fell from 14,861 MiB to 4,245 MiB and came back after the wake. The log line Sleep mode freed 10.37 GiB memory confirms the sleep.

Know the sleep mode problems at 0.31.0

Three problems with sleep mode are known at 0.31.0. We did not reproduce them ourselves:

• A level 2 wake does not reload a speculative draft. Issue 52479 reports that the weight reload covers only the target model. The target model then rejects every draft token, so decode falls to about the speed with no speculation. The output stays correct, and no error appears. The issue is open.
• A level 2 sleep clears calibrated KV scales. On compressed-tensors checkpoints with an FP8 KV cache, the first request after a reload of the weights returned NaN logprobs and a wrong first token. Release 0.32.0 fixes this.
• A weight update during sleep stops the engine. Issue 57853 reports a segfault when the trainer sends weights after /sleep and before /wake_up. The API server stays up, and every later request fails. The issue is open.

So after each level 2 wake, reload the weights, reset the prefix cache, and read the speculative acceptance counters. Always wake the server before the trainer sends weights.

Check the rollout logprobs

The trainer often compares the logprobs that vLLM returned with the logprobs it computes itself. The two never match exactly, because the kernels differ. A small, steady gap is normal. A gap that grows means the weights did not reach the generator. Track the gap every step. The reproducibility page explains why the same weights still give different numbers.

Before you read the gap, check which logprob vLLM returns. With top-p or top-k, the two values differ for most tokens.

Logprobs mode: the server option --logprobs-mode, which picks the value that vLLM returns for each token. The default is raw_logprobs, the logprob over the full vocabulary. processed_logprobs gives the logprob after the sampling settings change the distribution. Two more modes, raw_logits and processed_logits, return logits.

We checked both logprob modes on the second GPU of our L4 node on 2026-10-23. We used Qwen/Qwen3-0.6B in BF16, 16 prompts of 48 tokens, temperature 1, and top-p 0.9. A float32 Transformers pass over the same tokens gave the reference. That GPU is shared and has no clock lock, but the clock does not change these values.

engine and mode                                  returned minus reference, median   median absolute
vLLM --logprobs-mode raw_logprobs                0.0000                             0.005
vLLM --logprobs-mode processed_logprobs          +0.0527

• raw_logprobs matched the full-vocabulary logprob, with a median absolute gap of 0.005 from the kernels.
• processed_logprobs was higher by minus the log of the probability mass that top-p kept. The residual against that formula had a median of 8e-9.
• At top-p 0.9, the kept mass was below 0.99 for most sampled tokens, and the median shift was about 0.053.

So compare raw_logprobs with a trainer that scores over the full vocabulary. Compare processed_logprobs only with a trainer that scores over the same kept set.

Not measured on our hardware. We ran only that one level 1 check and the logprob check, both with small models. We have not timed a level 2 sleep or a weight update. Our Miles 101 manual measures the same loop with SGLang as the generator.

Your Qwen3 server returns a long block of thinking where you expected a short answer. When you finish this section, you can separate the thinking from the answer, turn the thinking off, and limit its length.

We did not measure the speed of reasoning models for this manual. We checked the fields and switches on this page on an L4 with Qwen3-8B-FP8. That covers the reasoning field, enable_thinking per request and per server, the thinking budget, and the stream order. The rest comes from the vLLM documentation and the source at 0.31.0.

Know what the model writes

Reasoning model: a model that writes its reasoning steps first, between markers such as <think> and </think>, and then the final answer. Qwen3, DeepSeek-R1, and QwQ are examples. Qwen3 thinks by default.

Without a reasoning parser, vLLM returns the whole output in content. The answer then starts with the thinking. The thinking also counts against max_tokens, so a short limit can end the request before the answer starts.

Turn on the parser

Start the server with the parser for your model family. For Qwen3, the parser is qwen3:

vllm serve Qwen/Qwen3-8B --reasoning-parser qwen3 2>&1 | tee server.log

The default is no parser. vLLM does not pick one from the model name. The vLLM documentation lists the parser name for each supported family, such as deepseek_r1 for DeepSeek-R1 and QwQ.

With the parser, a chat response has two text fields:

In a stream, each chunk carries the new text in delta.reasoning or in delta.content. Older releases called the field reasoning_content. At 0.31.0 the response field is reasoning, so update a client that reads the old name.

Turn the thinking off or limit it

You have three controls:

• For one request. Send "chat_template_kwargs": {"enable_thinking": false} in the request body. With the OpenAI client, put it in extra_body.
• For the whole server. Start the server with --default-chat-template-kwargs '{"enable_thinking": false}'. A request can still turn the thinking on again with its own chat_template_kwargs.
• A budget for the thinking. Send thinking_token_budget in the request. When the thinking reaches that many tokens, vLLM makes the model close the thinking and start the answer. The parser must be on.

The name enable_thinking comes from the Qwen3 chat template. Other families use other names, so read the chat template of your model.

Know what changes for JSON and tools

With a reasoning parser on, structured output applies to the answer only, after the thinking ends. The thinking itself is free text. The Qwen3 parser also works with tool calls, so a request can return reasoning and tool_calls together. For tool calls, also start the server with --enable-auto-tool-choice and a tool call parser, such as --tool-call-parser hermes for Qwen3. The JSON page shows how to ask for a schema.

Give a thinking request a large max_tokens. If the limit ends the request inside the thinking, content is empty and finish_reason is length. A response that a limit cut is not valid JSON.

Check the setup

Send one request with the thinking on and one with it off:

curl -s http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3-8B", "messages": [{"role": "user", "content": "Is 91 a prime number?"}], "max_tokens": 2048}'

The first reply has text in both reasoning and content. Add "chat_template_kwargs": {"enable_thinking": false} to the body and send it again. The second reply has an empty or missing reasoning field and a short content.

Your users wait for long answers at low load. When you finish this section, you can choose the draft depth for your concurrency.

Our result: on our clock-locked L4, EAGLE3 at depth 3 gave 1.6 times the baseline output at 32 users, on random prompts with 128-token outputs.

Turn it on

EAGLE3 adds a small draft model that guesses the next tokens from the hidden states of the target model. The target model checks the guesses in one pass, as the speculative decoding page explains. The target model keeps only the guesses it would have written itself.

You need a draft that someone trained for your target model. For Qwen/Qwen3-8B, we used Tengyunw/qwen3_8b_eagle3:

vllm serve Qwen/Qwen3-8B \
  --speculative-config '{"method": "eagle3", "model": "Tengyunw/qwen3_8b_eagle3", "num_speculative_tokens": 3}'

num_speculative_tokens is the draft depth: the number of tokens the draft guesses in each step. If you leave it out, vLLM takes the value from the draft config, if the config has one. Set it yourself, so you know what runs.

EAGLE3 keeps async scheduling on. The async scheduling page lists the methods that turn it off.

Know why the depth matters

A deeper draft can give more tokens for each pass of the target model. But each guess also costs work in the check. With one user, the GPU has spare compute in each step, so the extra guesses are almost free. With many users, each step is already full, and wrong guesses take compute from real tokens.

See what our runs show

We measured EAGLE3 at depths 1, 3, and 5 against no speculation on our L4 with Qwen/Qwen3-8B on 2026-10-22. This was the session locked at 900 MHz, and we ran 1 and 32 users with two seeds each. The prompts came from the random dataset of the benchmark tool, with 512 tokens in and 128 tokens out. Raw runs

At 32 users, depth 3 gave 1.6 times the output of no speculation. Its seed range did not overlap with any other config. Depth 5 gave less output than depth 3, so the extra guesses cost more than they gave.

At one user, depth 3 cut the time per token from 39.4 to 27.8 ms against depth 1. Depth 5 gave about the same as depth 3, and their seed ranges overlapped.

The run with no speculation at one user fell to 210 MHz, below the lock. A lock sets the highest clock, and it cannot stop the GPU from throttling. So we do not compare EAGLE3 against no speculation at one user.

We did not collect acceptance data from these runs, so we give no acceptance length here.

The median TPOT fell, but the gaps between the token groups grew. At 32 users, the median ITL rose from 84 ms with no speculation to 133 ms at depth 3. Each step now sends several tokens at once, so the stream arrives in larger and less frequent groups.

Know the limits of our result

Our gain holds for random prompts and outputs of 128 tokens. Other teams found that real traffic can give less. We did not measure these effects ourselves:

• The 2026 paper "SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding", by NVIDIA researchers, found that synthetic prompts overstate the speculative throughput of real prompts. It also found that the best draft depth changes with the batch size.
• The 2026 paper "Test-Time Speculation" studied EAGLE-3 and other drafts on MATH answers that ran toward 20,000 tokens. The acceptance length fell from about 3.7 to about 1.5 over those answers.

So a gain on short random answers does not predict a gain on long chat answers or on reasoning traces.

Read the acceptance over a long answer

We measured the acceptance length against the position in the answer on 2026-10-23. We used Qwen/Qwen3-8B-FP8 with an EAGLE3 draft and 3 speculative tokens, on the second GPU of our L4 node. Each of 8 prompts ran with thinking on, greedy, at one user, up to 4,096 tokens.

segment of the output   SGLang                 vLLM
0 to 256                2.05 to 2.64 (8)       2.14 to 2.51 (8)
256 to 512              1.95 to 2.64 (8)       2.09 to 2.55 (8)
512 to 1,024            2.11 to 2.77 (8)       2.12 to 2.79 (8)
1,024 to 2,048          2.09 to 2.47 (5)       1.72 to 2.66 (5)
2,048 to 4,096          2.33 and 2.48 (2)      2.35 and 2.60 (2)

Up to 4,096 tokens, the acceptance length did not fall steadily. It stayed at about 2.1 to 2.8 tokens for each step on both engines. Only 2 of the 8 prompts ran past 2,048 tokens, so the last segment rests on two answers. The GPU was shared and had no clock lock, which changes the speed but not the acceptance. Our limit of 4,096 tokens came from the memory we could use on that GPU. So we did not reach the range of the paper.

A second test on 2026-10-23 used Qwen/Qwen3-8B in BF16 on the first GPU of the node. It ran 8 users, with answers of up to 2,048 tokens and no ignore-eos. The median answer was 1,800 to 2,000 tokens. vLLM gave the acceptance from its counters before and after each run:

engine          dataset    mean acceptance length   accepted per draft position 1 / 2 / 3
SGLang EAGLE3   GSM8K      2.68
SGLang EAGLE3   ShareGPT   2.32
vLLM EAGLE3     GSM8K      2.69                     0.77 / 0.54 / 0.38
vLLM EAGLE3     ShareGPT   2.35                     0.67 / 0.42 / 0.26

The first draft token was accepted 77 percent of the time on GSM8K and 67 percent on ShareGPT, and each later position less. On short 128-token answers in the same session, SGLang gave 2.69 to 2.78 on GSM8K and 2.33 to 2.43 on ShareGPT. Its long answers stayed within 0.01 of those ranges. We have no short-answer acceptance for vLLM from this session, so the vLLM rows show only the level on long answers.

Read the acceptance on your own long answers before you trust a gain. The metrics at the end of this page give it.

Lock the clock before you compare

Our first EAGLE3 runs had no clock lock. At one user, the run with no speculation ran at 210 MHz, and the EAGLE3 run at about 1,400 MHz. A gap that large hides the effect of speculation, so we set those runs aside.

Before you compare speculative decoding with no speculation, lock the clock or record it for each run. Compare two runs only at the same clock. The lock needs root:

sudo nvidia-smi -lgc 900,900
sudo nvidia-smi -rgc

The first command locks the SM clock at 900 MHz, and the second releases the lock after the runs.

Choose the depth

• For Qwen3-8B with this draft, start with a depth of 3. On our L4 it gave the best output at 32 users and tied at one user.
• For a busy server, sweep the depth on your own model. On our L4, depth 5 lost output at 32 users.
• If your load changes a lot through the day, vLLM can set the depth by batch size with num_speculative_tokens_per_batch_size. We did not test it.

Check the gain on your prompts

Run a sample of your own traffic with and without the config. Keep your real output lengths, and do not use --ignore-eos. Compare these numbers:

1. The acceptance rate, from the two counters below.
2. The median TPOT and the output throughput.
3. The P99 ITL and the P99 TTFT, because a gain in the median can hide a loss in the tail.
4. The acceptance rate of your longest answers alone, if your answers run to thousands of tokens.

Read the two counters from /metrics:

curl -s http://localhost:8000/metrics | grep -E "spec_decode_num_(draft|accepted)_tokens"

Divide the accepted tokens by the draft tokens. vLLM also writes a SpecDecoding metrics line to the log, with the mean acceptance length.

Your users ask for edits of documents they send. When you finish this section, you can decide when n-gram speculation beats a draft model.

Our result: on our L4, n-gram cut the time per token from 242 to 77 ms for one user, on random prompts with 128-token outputs.

Turn it on

N-gram speculation guesses the next tokens from text that the request already holds. It takes the last few tokens of the output and finds the same tokens earlier in the request. Then it proposes the tokens that came after them. The target model then checks the guesses in one pass, as the speculative decoding page explains.

It needs no draft model, so it uses no extra GPU memory for weights. Start the server with a speculative config:

vllm serve Qwen/Qwen3-8B \
  --speculative-config '{"method": "ngram", "num_speculative_tokens": 3, "prompt_lookup_max": 4}'

num_speculative_tokens is the number of guesses in each step. prompt_lookup_max is the longest run of tokens that the proposer tries to match. Our runs used these two values.

Know when it pays

The proposer can only guess text that already appears in the request. Work that copies the prompt gives it many matches:

• an edit of a document that the user sends
• a summary that quotes the source
• a code change that keeps most lines
• extraction of fields from a long input

Open chat copies little. For that work, most guesses fail, and each step writes one token, as it would without speculation.

See what our run shows

We measured the time per token with and without n-gram speculation in the shared-GPU session of our L4 node, with Qwen/Qwen3-8B, on 2026-10-22. That GPU held a pool of 2,928 tokens, so we tested 1 and 4 users with two seeds each. The prompts came from the random dataset of the benchmark tool, with 512 tokens in and 128 tokens out. Raw runs

The GPU ran at 375 to 443 MHz, far below its full clock, because the GPU is shared and throttled. The times are slow for that reason. Read the ratio between the two columns, not the absolute times. The n-gram runs had a clock about 10 percent higher, which is small next to the threefold gain.

At one user, output rose by 86 percent. At four users it rose by 48 percent, and the worst token gap grew from 290 to 1,691 ms. We did not trace the cause of that gap.

We did not record the acceptance counters, so we cannot say how many guesses matched. Treat our gain as a result for random prompts and 128-token outputs only. On your own prompts, the gain depends on how much the output copies the input.

Another team reports the same limit, and we did not measure it ourselves. The paper "Speculative Decoding: Performance or Illusion?", posted at the end of 2025, studied speculation in vLLM. It found that the acceptance length changes a lot between requests and between datasets. Our own run also shows that a lower median can hide a worse tail. At four users the median time per token fell, and the P99 ITL rose.

Know what it costs

• N-gram speculation turns off async scheduling, with only a warning in the log. The async scheduling page lists the methods that do this.
• It runs on the older model runner, not on Model Runner V2.

Check the gain on your prompts

Run a sample of your own traffic with and without the config. Keep your real output lengths, and do not use --ignore-eos. Compare the median TPOT, the P99 ITL, and the P99 TTFT, not only the median. Then read two counters from /metrics:

curl -s http://localhost:8000/metrics | grep -E "spec_decode_num_(draft|accepted)_tokens"

Divide the accepted tokens by the draft tokens. A high share means that your output copies the prompt and the method pays. A low share means that a draft model such as EAGLE3 is the better test.

Startup takes minutes and you want it faster. When you finish this section, you can decide what each compile level and eager mode cost.

Our result: on our L4, eager mode cost about 2 percent of the time per token in a cooled run. In a throttled run, -O0 cost about 2 percent for one user, and -O3 matched the default.

Know the switches

vLLM sets compile and CUDA graphs with one level, from -O0 to -O3. The default is -O2. The compile page explains what each part does.

Read what the start costs

The startup log gives the time of each part. On our DGX Spark, with Qwen3-30B-A3B, the two graph captures and the compile took about 96 seconds of a start of about eight minutes:

(EngineCore pid=143) INFO 10-21 10:59:45 [model_runner.py:1087] Graph capturing finished in 39 secs, took 1.17 GiB
(EngineCore pid=143) INFO 10-21 11:00:51 [model_runner.py:1087] Graph capturing finished in 38 secs, took 0.31 GiB
(EngineCore pid=143) INFO 10-21 11:00:53 [core.py:373] init engine (profile, create kv cache, warmup model) took 127.54 s (compilation: 18.67 s)

On that start, loading the weights took about 330 seconds, much more than the compile. A later start can read the compile cache, so the compile share gets smaller.

See what our runs show

We measured eager mode against the default on the L4 with Qwen/Qwen3-8B on 2026-10-21, in the full-clock session. Both runs used the same pool, seven concurrencies, and three seeds, with the GPU cooled before each run. Raw runs

The seed ranges did not overlap from 1 to 32 users, so the gap is real, and it is small. On our DGX Spark, eager mode cost 2 to 6 percent of the time per token.

We then ran the compile levels on 2026-10-22, in the throttled session, at 1 and 32 users. The default had three seeds and the other two levels had two. The GPU was throttled for all three: the median clock was 210 MHz at one user and 450 to 548 MHz at 32 users.

At one user, -O0 was 2.2 percent slower, and the seed ranges did not overlap. At 32 users, the three levels overlapped. -O3 matched the default, as the code says it should at this version. Because of the low clock, read these as a comparison inside one session, not as speeds you will see.

Decide

Keep the default for production. The graphs and the compile cost about one and a half minutes on our DGX Spark start, and the cache makes later starts faster.

When you test a change and restart often, use -O0. It skips the compile and the graph capture, and it cost us about 2 percent of decode speed. Use --enforce-eager only to debug a kernel, because it also skips the warmup, so the first requests after a start are slow.

You read about async scheduling and you want to know whether to set it. When you finish this section, you can check whether your server runs with it, and name the settings that turn it off.

Our result: on a throttled L4, turning async scheduling off made one user about 3 percent slower per token. Treat it as an unconfirmed result, because the clock was low and we had two seeds.

Know what it does

Without async scheduling, each step runs in order. The scheduler picks a batch, the GPU runs it, and the scheduler reads the result before it picks the next batch. While the scheduler works, the GPU waits.

With async scheduling, the scheduler picks the next batch while the GPU still runs the current one. It counts the tokens that are not sampled yet as placeholders, so it does not schedule a request past its token limit. The GPU spends less time idle between steps. The Model Runner V2 page explains why the new model runner makes this possible.

Know when it is on

In vLLM 0.31.0, the setting has three values. --async-scheduling turns it on, --no-async-scheduling turns it off, and no flag lets vLLM decide. When you set no flag, vLLM turns it on unless, among other cases, one of these holds:

• The model is a pooling model, such as an embedding model.
• The speculation method is n-gram, suffix, Medusa, or MLP speculator. EAGLE, MTP, DFlash, and DSpark methods, the GPU n-gram method, and a draft model keep it on.
• Pipeline parallelism is on and the old model runner runs.
• The executor does not support it.

In those cases, vLLM turns it off. For most of them it writes a warning to the log, but for a pooling model only a debug line. The server runs, so the change is easy to miss. Read the startup log after you add a speculation method or change the parallel layout.

See how much two runs of one setting differ

We ran the L4 load sweep twice on 2026-10-21 with Qwen/Qwen3-8B, in the full-clock session. One run had no flag, and one had --async-scheduling. For this model with no speculation, the flag sets what the default already sets, so the two runs are the same setting.

The two runs differed by at most 1.8 percent of output and 0.8 percent of time per token. Their seed ranges overlapped at every concurrency. So on this card, a gap of a few percent between two configs is run-to-run variation unless the seed ranges separate. The lab notes have the full table under the configs vllm-b and vllm-async.

See what turning it off costs

We measured --no-async-scheduling against the default in the shared-GPU session of our L4 node on 2026-10-22, with Qwen/Qwen3-8B. That GPU held a pool of 2,928 tokens, so we tested 1 and 4 users. Raw runs

At one user, the time per token rose by 3.3 percent with async scheduling off, and the seed ranges did not overlap. That gap is larger than the 0.8 percent that two runs of one setting differed by. But we measured that variation on the other GPU at full clocks. At four users, it rose by 1.7 percent, from one seed only.

The GPU ran at 368 to 450 MHz, far below its full clock. The runs with async scheduling off at one user had a clock 2 percent lower, close to the size of the effect. So read this as an unconfirmed result. The gain is small at these batch sizes, and we did not test it at a full clock or at high concurrency.

Decide

Set no flag, and let vLLM choose. If you add a speculation method or pipeline parallelism, read the log for the warning. If the warning appears, measure the time per token with and without the feature, because you lose async scheduling together with the feature.

A forum post says a different attention backend is faster. When you finish this section, you can read the backend vLLM picked and decide when to override it.

Our result: on our L4, a forced Triton backend fixed a start that failed in the backend vLLM picked.

Read the backend vLLM picked

vLLM writes one line at startup that names the backend and the other backends that could run. Search the log for it:

grep "attention backend" server.log

On our DGX Spark, with Qwen3-30B-A3B and the default settings, the line was:

(EngineCore pid=143) INFO 10-21 10:53:17 [cuda.py:526] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].

Know the order vLLM tries

vLLM has one list of backends for each GPU class. It removes each backend that cannot run with your model and your settings, and it picks the first one that is left. For a standard model in vLLM 0.31.0, the order is:

The reasons for each removal go to the debug log, not to the info log. To see them, compare the list in the startup line with the table. A backend that is missing from the list was removed.

Know the switch that an FP8 KV cache makes

In vLLM 0.31.0, FlashAttention reads an FP8 KV cache only on SM 9.0 and on the SM 10.x family. So on an L4 or a DGX Spark, --kv-cache-dtype fp8 changes the backend with no warning. The FP8 KV cache page tells what happened on our L4.

Force a backend

Set the backend with --attention-backend:

vllm serve Qwen/Qwen3-8B --kv-cache-dtype fp8 --attention-backend TRITON_ATTN

If the backend you set cannot run with your model and settings, vLLM stops at startup with an error that gives the reason. It does not move to another backend. So a forced backend is safe to try: it runs, or it fails at once.

--block-size also removes backends. If it removes a backend that is higher in the list, vLLM writes a warning that names it.

See what our run shows

Our only forced-backend run changed the backend and the KV data type together, so its result does not show which backend is faster. We have not compared backends with one KV data type on any of our GPUs.

Decide

Leave the backend to vLLM. Override it for one of two reasons:

• The start fails in the backend that vLLM picked, as FlashInfer did on our L4.
• You measured a slowdown on your own load, and a different backend fixed it on the same load.

A forum post that says a backend is faster tested a different GPU, model, or load. Measure on yours before you change it.

You changed a flag and you want to know if it helped. When you finish this section, you can run a concurrency sweep, find the latency jump, and avoid the four mistakes that make two runs disagree.

Our result: on our L4, output peaked near 275 tokens per second at 32 users and fell at 64 users.

Run one sweep

Our tables came from the benchmark client of SGLang 0.5.22, not from the vLLM client, at temperature 0. We used one client for both engines, so that the engine page compares the engines with one method. To compare your GPU with our table, use the same client. Install it in a separate virtual environment, so that it does not change your vLLM install:

uv venv bench --python 3.12 --seed
source bench/bin/activate
uv pip install sglang==0.5.22

Start the vLLM server in one terminal. In a second terminal, run the client once for each concurrency. This is the command that made our table:

for c in 1 2 4 8 16 32 64; do
  python -m sglang.benchmark.serving --backend vllm --port 8000 \
    --dataset-name random --random-input-len 512 --random-output-len 128 --random-range-ratio 1.0 \
    --max-concurrency $c --num-prompts $((c * 4 > 16 ? c * 4 : 16)) --seed 1 \
    --output-file sweep-c$c.jsonl --output-details
done

We ran the loop three times, with --seed 1, 2, and 3, and report the median. With --backend vllm, the client sends each request to /v1/completions. It sends one warmup request first, and it tells the server to ignore the end-of-sequence token, so every answer has the full 128 tokens.

Each run prints a summary with four numbers. They are the output throughput, the time to first token (TTFT), the time per output token (TPOT), and the inter-token latency (ITL). Keep the four numbers together. A throughput number alone hides the latency that your users feel.

Read the knee

Knee: the concurrency where the latency starts to rise fast while the output stops rising. Past the knee, more users get slower answers and the server writes almost no more tokens.

Here is the sweep from our L4 with Qwen/Qwen3-8B in BF16, run on 2026-10-21 in the full-clock session. Each number is the median of three seeds. The pool was 24,096 tokens.

Up to 16 users, each new user adds output, and the time per token rises only from 60 to 70 ms. In this table, the median first token goes from 0.3 s to 1.8 s between 16 and 32 users. The P99 ITL goes from 85 to 613 ms.

Do not take that jump as the knee. This sweep did not reset the prefix cache between user counts, and each seed sent the same first prompts at every count. Below 32 users, many requests found their prompt in the cache. We ran the sweep again on 2026-10-23 with a reset before each run, with the clock locked at 900 MHz:

median TTFT, flushed against unflushed
users   vLLM                                   SGLang
4       650 / 145 ms (4.5x)                    532 to 645 / 141 ms (3.8 to 4.6x)
8       951 to 978 / 264 to 274 ms (3.6x)      1,174 to 1,273 / 173 to 178 ms (6.6 to 7.3x)
16      1,550 to 1,587 / 294 to 295 ms (5.3x)  1,804 to 1,944 / 218 ms (8.3 to 8.9x)
32      1,982 to 2,084 / 2,012 to 3,091 ms     3,216 to 3,228 / 2,710 to 2,917 ms
…

With the reset, the median first token rose at every step. It was about 0.65 s at 4 users, 1.0 s at 8, and 1.6 s at 16. The P99 ITL jumped from about 70 ms at 8 users to 522 ms at 16. So on this card the knee sits between 8 and 16 users. The cache hid it. Pick your production concurrency below the knee that you measure with a reset.

At 64 users, the output fell. Two causes overlap here. At 640 tokens each, 64 users need more than the 24,096-token pool, so requests waited. The GPU also throttled, with a median clock of 1,005 MHz against 1,455 at 16 users. Our sweep cannot separate the two.

Avoid the four mistakes

1. Reusing a warm cache. With the random dataset and a fixed seed, a second run sends the same prompts. The prefix cache then serves part of them, and the second run looks faster. The next section gives the protocol.
2. Letting the model stop early. Without --ignore-eos, some answers end before the output length. The run then measures fewer tokens than you asked for.
3. Skipping the warmup. The first requests after a start pay for compiled kernels and cold caches. Use --num-warmups or send a few requests before you measure.
4. Ignoring the clock. A GPU that throttles or that other work shares runs slower. Record nvidia-smi --query-gpu=clocks.sm,temperature.gpu --format=csv -l 1 during the run. Compare two runs only at similar clocks.

Reset the prefix cache between runs

The prefix cache keeps the KV blocks of earlier prompts. It changes two things between runs: the speed, and sometimes the output of a greedy request. The reproducibility page shows the output effect.

To see the speed effect, we ran one load twice with the same seed and no reset. This was on the second GPU of our L4 node on 2026-10-22 and 2026-10-23. We used Qwen/Qwen3-0.6B, 64 prompts at 16 users, 512 tokens in and 128 out. That GPU is shared with its owner and had no clock lock, so trust the ratio and not the times.

engine        fresh   repeat, not flushed   repeat after a flush   repeat, not flushed
…
vLLM 0.6B     925     148                   1,163                  184

The repeat without a reset read a median TTFT about 6 times lower than the fresh run. After a reset, the TTFT came back to the level of the fresh run. The time per token changed by less than 15 percent, because a cache hit removes prefill work and leaves the decode work the same.

Use this protocol for every run that you compare:

1. Start each run with an empty cache. Restart the server, or call POST /reset_prefix_cache. That route exists only with VLLM_SERVER_DEV_MODE=1, and a server in that mode must not be exposed.
2. Use a new --seed for each run, so that the random dataset sends new prompts.
3. Record the cached tokens. Read vllm:prefix_cache_hits_total and vllm:prefix_cache_queries_total before and after the run.
4. For a count on each response, start the server with --enable-prompt-tokens-details. Then read usage.prompt_tokens_details.cached_tokens in each response.
5. Write the cache setting and the cached tokens next to the result.

vllm bench sweep serve resets the server caches between its runs for you.

Choose a closed loop or a fixed rate

--max-concurrency keeps a fixed number of users busy. When the server slows down, the users send their next request later, so the load drops by itself. This closed loop never falls behind.

--request-rate sends requests at a fixed rate, whether the server keeps up or not. Past the rate the server can serve, the queue grows, and the first token takes longer with every request. Real traffic behaves more like a fixed rate. Run both before you set a capacity limit.

Use the vLLM client

vLLM has its own client, vllm bench serve. It reports the same four numbers, and it needs no second install. At 0.31.0, VLLM_USE_RUST_BENCH=1 runs a Rust version of the same client. This loop runs the same sweep with it:

for c in 1 2 4 8 16 32 64; do
  vllm bench serve --model Qwen/Qwen3-8B \
    --dataset-name random --random-input-len 512 --random-output-len 128 \
    --max-concurrency $c --num-prompts $((c * 4 > 16 ? c * 4 : 16)) \
    --seed $c --ignore-eos --num-warmups 2 \
    --save-result --result-filename sweep-c$c.json
done

We ran this loop as written on 2026-10-23, with Qwen/Qwen3-8B-FP8 on the shared second GPU of our L4 node. Each run finished and wrote its result file. The GPU was throttled, so we give no numbers from that run.

We ran three clients against one vLLM server with Qwen/Qwen3-0.6B on 2026-10-23: the SGLang client, vllm bench serve, and its Rust version. This was the shared second GPU of our L4 node. Its clock was throttled, but it stayed at 420 to 495 MHz for every run. We reset the cache before every run and alternated the clients over two rounds.

users   temperature   SGLang client    vllm bench       vllm bench, Rust
…
64      0             359, 348         379, 377         371, 365
64      0.7           331, 343         360, 363         356, 353

• The two vLLM clients agreed. The Python and the Rust versions of vllm bench serve gave the same numbers within the noise at 1, 16, and 64 users.
• The SGLang client read a slower server. Its median TTFT was 3 to 10 percent higher than vllm bench serve. At 16 and 64 users, its output throughput was 2 to 8 percent lower. At one user, one of its rounds read a higher throughput than every other round, so we give no throughput gap there.
• The prompts differ. The two clients send different prompt tokens from their random datasets, so part of a small gap can come from the prompts.
• Temperature changes the result. Temperature 0.7 raised the median TPOT over temperature 0. It was 10 to 13 percent higher at one user, and 3 to 6 percent higher at 16 and 64 users.

So our tables, from the SGLang client at temperature 0, can read a few percent slower than a run with vllm bench serve. Pick one client and one temperature, and keep them for every run that you compare.

Your server is in production and you must know when it is in trouble. When you finish this section, you can pick the metrics to alert on.

Scrape the endpoint

vLLM exports Prometheus metrics at /metrics on the same port as the API. No flag is needed. Add a job like this one to the scrape_configs list of your Prometheus config, and change the target to your server:

scrape_configs:
  - job_name: vllm
    scrape_interval: 15s
    metrics_path: /metrics
    static_configs:
      - targets: ["vllm-host:8000"]

We did not run Prometheus against our servers.

Alert on five metrics

Set the latency targets from your product, not from the defaults of a dashboard. Our sweeps used 1,000 ms for the first token and 100 ms for each output token.

The two latency metrics are histograms, so the 99th percentile comes from a PromQL query. This expression is above 0 when the P99 of the first-token time over the last five minutes is more than 1 second:

histogram_quantile(0.99, sum by (le) (rate(vllm:time_to_first_token_seconds_bucket[5m]))) > 1

Use the same form with vllm:inter_token_latency_seconds_bucket and a target of 0.1 for the token gaps.

Read them together

One metric alone misleads. Read them in pairs:

• Waiting above 0 with the cache usage below 0.9. The pool has room, but the batch is full. --max-num-seqs or --max-num-batched-tokens limits the admission.
• Waiting above 0 with the cache usage near 1. The pool limits the admission. Add memory, or serve fewer users on this GPU.
• A slow first token with no waiting requests. The prompts are long, or the tokenizer in the API server is slow. Compare the queue time, vllm:request_queue_time_seconds, with the first-token time.
• Preemptions with long answers. Running requests grow until the pool runs out. Lower the concurrency, or use an FP8 KV cache.

Watch the cache and speculation

Two more pairs of counters tell you whether a feature pays:

• The prefix cache hits divided by the queries give the hit rate. A rate that falls after a release of your application often means a change at the start of the prompt.
• With speculative decoding, the accepted tokens divided by the draft tokens give the acceptance rate. A falling rate means the proposer guesses badly on the current traffic.

Watch the GPU too

The vLLM metrics do not show the GPU clock or temperature. A hot or shared GPU runs slower, and every latency rises with no change in the vLLM metrics. Export nvidia-smi data or use the NVIDIA DCGM exporter next to the vLLM metrics.

Not measured on our hardware. Our sweeps did not scrape /metrics. Our run with a small pool made requests wait in the queue, but we did not save its log or its metrics. The chat page shows that run.

You are choosing a serving engine and you want a measurement, not a reputation. When you finish this section, you can match your traffic to one of our side-by-side runs and say how far the result goes.

Our result: on our L4, throughput matched within 3 percent up to 32 users. On our DGX Spark, the worst token gap of vLLM was about twice that of SGLang from 32 users.

Know what we compared

We ran vLLM 0.31.0 and SGLang 0.5.22 with their defaults on two machines:

• An NVIDIA L4 with Qwen/Qwen3-8B in BF16, in the full-clock session. Both pools held about 24,100 tokens. The load runs were on 2026-10-17, and the rate and waves runs on 2026-10-18.
• A DGX Spark with Qwen/Qwen3-30B-A3B-Instruct-2507 in BF16. The pools held 234,960 tokens for vLLM and between 233,303 and 251,910 tokens for SGLang. Each SGLang run was on the same day as its vLLM run, 2026-10-21.

Each point is the median of three seeds, from one benchmark client for both engines. This is two models on two cards. Treat the results as a guide to what you should measure, not as a rule.

Read the result for your traffic

See the numbers behind each row

Many users at once, L4. At 32 users, vLLM gave 273.9 tok/s and SGLang 271.3. The median first token took 1,813 ms on vLLM and 2,871 ms on SGLang. The P99 ITL was 619 ms on vLLM and 102 ms on SGLang. At 64 users the output differed by 6 percent. The vLLM run had a lower median clock, 1,013 against 1,215 MHz, so we do not count that gap.

These runs did not reset the prefix cache. We ran them again with a reset at 900 MHz on 2026-10-23, and the direction held. At 32 users the median first token was 2.0 to 2.1 s on vLLM and 3.2 s on SGLang. The P99 ITL was 668 to 964 ms on vLLM and 105 to 106 ms on SGLang. At 16 users it was 522 ms against 82 ms.

A fixed rate, L4. At 4 requests per second, vLLM gave 286.6 tok/s and SGLang 257.5. The seed ranges did not overlap. At 8 requests per second the ranges overlapped. The runs at 16 requests per second ran below our clock floor, the lowest clock we accept for a run, so we left them out.

Mixed and equal lengths, both machines. Our waves workload sent 64 prompts at 16 users, with a length ratio that sets how much the prompt and output lengths vary. The lab notes define it. With mixed lengths on the L4, the P99 ITL was 191 ms on vLLM and 363 ms on SGLang. With equal lengths it was 502 ms on vLLM and 81 ms on SGLang. The DGX Spark showed the same direction: 188 against 198 ms with mixed lengths, and 394 against 159 ms with equal lengths.

Mixture of experts, DGX Spark. At 64 users, vLLM gave 239.3 tok/s and SGLang 245.1, inside the seed spread. The P99 ITL was 528 ms on vLLM and 271 ms on SGLang, and 478 against 220 ms at 32 users. The seed ranges did not overlap. The median first token was also longer on vLLM at every load from 2 users.

Long prompts at a low rate, DGX Spark. With prompts of about 1,000 to 4,000 tokens at 0.5 requests per second, vLLM had the lower P99 ITL, 571 against 922 ms. At 1 and 2 requests per second, SGLang had the lower one. For this sweep the SGLang pool was 7 percent larger than the vLLM pool, so read this row with care.

Decide

Pick on the things that do not change with tuning first:

1. Model support. Check that your model and its quantization run on the engine and the version you pin.
2. Startup time. On our DGX Spark, vLLM took 330 seconds to load the 57 GiB of weights.
3. Your hardware. A result on an L4 or a GB10 does not carry over to a datacenter GPU without a test.

Then run the sweep on your own traffic with both engines. Use one benchmark client and equal pools.

One GPU is not enough. When you finish this section, you can pick the parallel layout from a short decision tree.

Answer three questions

When the model needs more than one machine:

• Set --pipeline-parallel-size to the number of machines, and --tensor-parallel-size to the GPUs in each machine.
• The tensor-parallel size must divide the attention-head count of the model.
• If the GPUs in each machine do not divide it, use --tensor-parallel-size 1 and set the pipeline size to the total GPU count.

Know what each layout costs

Tensor parallelism also splits the weights and the KV cache, so each GPU has more room for the cache. When the weights fill most of one GPU, two GPUs can hold several times the KV pool of one.

Check the link between GPUs

Tensor parallelism sends data after every layer. On GPUs joined only by PCIe, such as two L4s in one host, that traffic can cost more than the split saves. The vLLM documentation suggests pipeline parallelism on such a machine, because it sends data only at stage boundaries. Pipeline parallelism also accepts a GPU count that does not divide the model evenly. Measure the layouts against each other on such a machine before you pick.

Not measured on our hardware. We have not run vLLM on two L4s with tensor parallelism or data parallelism.

Start a second machine

vLLM runs one server across machines with Ray, the default for more than one machine, or with its own multiprocess launcher. We did not run either. The steps below come from the vLLM 0.31.0 documentation, for two machines with 8 GPUs each.

First, check that the GPUs can talk to each other. The vLLM troubleshooting guide has a short test script that runs NCCL and Gloo all-reduce calls. Save it as test.py on each machine. Run it on both machines at once, with NODE_RANK=0 on the first machine and NODE_RANK=1 on the second:

NCCL_DEBUG=TRACE torchrun --nnodes 2 --nproc-per-node 8 \
  --rdzv_backend=static --rdzv_endpoint=<head-ip>:29400 \
  --node-rank $NODE_RANK test.py

When the network and the drivers work, each process prints four success lines, the first of them PyTorch NCCL is successful!. The guide text names a different line, so trust the lines that the script prints.

With Ray. The vLLM repository has a run_cluster.sh script that starts the vLLM image and a Ray node on each machine. Give each machine its own VLLM_HOST_IP, on a private network. On the first machine, run:

bash run_cluster.sh vllm/vllm-openai:v0.31.0 <head-ip> --head ~/.cache/huggingface -e VLLM_HOST_IP=<head-ip>

On the second machine, run:

bash run_cluster.sh vllm/vllm-openai:v0.31.0 <head-ip> --worker ~/.cache/huggingface -e VLLM_HOST_IP=<worker-ip>

Keep both shells open, because closing one stops the cluster. Then, inside a container, check that Ray sees every node and GPU:

ray status

Start the server on one machine only. It uses the GPUs of every machine:

vllm serve Qwen/Qwen3-30B-A3B-Instruct-2507 \
  --tensor-parallel-size 8 --pipeline-parallel-size 2 \
  --distributed-executor-backend ray

Without Ray. Run vllm serve on both machines with the same model and sizes. Add --nnodes 2 --node-rank 0 --master-addr <head-ip> on the first machine. Add --nnodes 2 --node-rank 1 --master-addr <head-ip> --headless on the second machine, which then runs workers and no API server.

The traffic between the machines is not encrypted, and it can run code on the other side. Keep it on a private network that untrusted machines cannot reach.

Count the processes

With data parallelism, vLLM starts one engine core for each copy, and one coordinator that keeps them in step. With tensor parallelism, it starts one worker for each GPU under one engine core. The process page gives the count.

Use context parallelism for very long prompts

For prompts of hundreds of thousands of tokens, vLLM can split one sequence across GPUs with prefill and decode context parallelism. Use it only when one long request is the problem, not when many short ones are.

You serve a mixture-of-experts model. When you finish this section, you can decide when expert parallelism helps.

Our result: one DGX Spark served Qwen3-30B-A3B without expert parallelism, at 239 tokens per second for 64 users.

Know what changes with a MoE model

A mixture-of-experts layer has many experts, and each token uses only a few of them. Qwen/Qwen3-30B-A3B-Instruct-2507 has 30 billion parameters, and about 3 billion are active for each token. So the model needs memory for all its weights, but each token costs compute for only a small part.

Start with one device if the model fits

On our DGX Spark, one GB10 held the whole model in BF16. The weights took 56.93 GiB, and the server gave 239.3 tok/s at 64 users. One device needs no communication between GPUs. If the model and its KV cache fit, start there.

Our reading: a batch of many tokens uses many different experts, so a decode step reads more of the weights as the batch grows. On a device with limited memory bandwidth, that makes the time per token grow with the batch. On our GB10, the median time per token went from 32 ms at one user to 244 ms at 64 users.

Add expert parallelism for large models

On one machine with eight GPUs, spread the experts across all of them:

vllm serve <model> --data-parallel-size 8 --enable-expert-parallel

For a model that does not fit one machine, such as the large DeepSeek models, run one command on each machine. This example uses two machines with eight GPUs each. First install DeepEP and DeepGEMM on both machines, as the expert parallel guide in the vLLM documentation describes. They are not part of the normal install, and a server with a DeepEP backend stops at startup without DeepEP. The first machine takes the requests:

vllm serve <model> --enable-expert-parallel \
  --all2all-backend deepep_low_latency \
  --data-parallel-size 16 --data-parallel-size-local 8 \
  --data-parallel-address 192.168.1.100 --data-parallel-rpc-port 13345

The second machine runs headless, with no API server. Its start rank is the count of ranks on the machines before it:

vllm serve <model> --enable-expert-parallel \
  --all2all-backend deepep_low_latency \
  --data-parallel-size 16 --data-parallel-size-local 8 \
  --data-parallel-start-rank 8 \
  --data-parallel-address 192.168.1.100 --data-parallel-rpc-port 13345 \
  --headless

Set the address to the IP of the first machine, and use the same RPC port on both. Across machines, the vLLM documentation uses the DeepEP kernels for the exchange of tokens.

With expert parallelism, each GPU holds some of the experts, and tokens travel to the GPU that holds their expert and back. The attention layers run in data parallel on each GPU. The all-to-all exchange of tokens needs fast links between the GPUs.

Two more features help at large scale:

• Expert load balancing. --enable-eplb moves busy experts to spread the load, because some experts receive far more tokens than others.
• Elastic expert parallelism. It adds or removes GPUs from a running deployment. Release 0.32 brought it to Model Runner V2.

Not measured on our hardware. We have one GPU on each machine, so we have not run expert parallelism.

Check the setup

On a multi-GPU deployment, watch the time per token as the load grows, and compare it with the same model on fewer GPUs. If the time per token rises faster than the output, the exchange between GPUs limits the step.

Long prompts slow the stream for every user. When you finish this section, you can decide when disaggregated serving pays.

Know the problem it solves

On one server, a long prompt shares each step with the running streams. While it computes, every stream waits a little longer for its next token. With many long prompts, the worst token gaps grow for everyone.

Disaggregated serving runs two kinds of server. A prefill server computes the prompt and builds its KV cache. A decode server receives that cache and generates the answer. Long prompts then never share a step with a stream.

Know what it does not solve

The vLLM documentation says that disaggregated prefill does not raise throughput. It gives you control over the first token and the stream as two separate numbers, and it controls the tail of the inter-token latency. It costs a transfer of the KV cache for every request, and more servers to run.

Even for the tail, the gain is a condition, not a default. The paper "Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implications" compared split servers with servers that run both stages. It found that the gain is not guaranteed and depends on the request load and on the path that moves the KV cache. We did not measure this ourselves.

Pick it when the 99th percentile of the inter-token latency matters and long prompts cause it. Do not pick it to serve more users on the same GPUs. Before you keep it, compare it with the same GPUs run as plain servers, at your real load.

Set it up

Each server gets a KV connector and a role in --kv-transfer-config. The NIXL connector is the common choice. It needs the NIXL library, which the vLLM wheel does not install. Install it in the same environment as vLLM first. The vLLM requirements at 0.31.0 pin version 1.4.1:

uv pip install nixl==1.4.1

The prefill server produces the cache, and the decode server consumes it. On one host with two GPUs, give each server its own GPU, its own port, and its own NIXL side channel port:

CUDA_VISIBLE_DEVICES=0 VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \
vllm serve Qwen/Qwen3-8B --port 8100 \
  --kv-transfer-config '{"kv_connector": "NixlConnector", "kv_role": "kv_producer"}'

CUDA_VISIBLE_DEVICES=1 VLLM_NIXL_SIDE_CHANNEL_PORT=5601 \
vllm serve Qwen/Qwen3-8B --port 8200 \
  --kv-transfer-config '{"kv_connector": "NixlConnector", "kv_role": "kv_consumer"}'

Each server writes Creating v1 connector with name: NixlConnector and Initializing NIXL wrapper in its startup log. If the log shows NIXL is not available, the library is missing from the environment.

Then put a router in front. For each request, the router sends the request to the prefill server first and reads the KV transfer parameters from its response. It then sends the request to the decode server with those parameters, so that the decode server can load the cache. The vLLM repository has example scripts and a proxy for this setup. Its own CI tests it on pairs of L4 GPUs.

If a cache transfer fails, the default policy fails the request. The kv_load_failure_policy field set to recompute computes the prompt again on the decode server instead.

Not measured on our hardware. We have not run disaggregated serving with vLLM on two L4s.

Size the two sides

The ratio of prefill to decode servers depends on your prompt and answer lengths. Long prompts with short answers need more prefill capacity. Short prompts with long answers need more decode capacity. Measure the first token and the stream on each side, and move one server at a time.

You must run many vLLM servers on a cluster. When you finish this section, you can pick the layer that runs vLLM on Kubernetes.

Start with what one Deployment gives you

The vLLM documentation shows a plain Kubernetes setup. It has a volume for the weights, a secret for the Hugging Face token, a Deployment that runs vllm serve, and a Service. A small Helm chart in the vLLM repository packages the same pieces. It adds a job that downloads the model and an optional Kubernetes autoscaler.

Here is a minimal version in our own form, for one GPU and one model. We did not run it on a cluster. It needs a cluster where the NVIDIA device plugin offers nvidia.com/gpu. For a model with restricted access, add a Secret named hf-token with a token key. Create a Secret named vllm-api-key with a key key, so that the /v1 routes need the key.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-qwen3-8b
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-qwen3-8b
  template:
    metadata:
      labels:
        app: vllm-qwen3-8b
    spec:
      terminationGracePeriodSeconds: 120
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.31.0
          args: ["Qwen/Qwen3-8B", "--max-model-len", "8192", "--reasoning-parser", "qwen3"]
          ports:
            - containerPort: 8000
          env:
            - name: HF_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-token
                  key: token
                  optional: true
            - name: VLLM_API_KEY
              valueFrom:
                secretKeyRef:
                  name: vllm-api-key
                  key: key
          resources:
            limits:
              nvidia.com/gpu: 1
          startupProbe:
            httpGet:
              path: /health
              port: 8000
            periodSeconds: 10
            failureThreshold: 90
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            periodSeconds: 30
            failureThreshold: 3
          volumeMounts:
            - name: hf-cache
              mountPath: /root/.cache/huggingface
            - name: shm
              mountPath: /dev/shm
      volumes:
        - name: hf-cache
          emptyDir: {}
        - name: shm
          emptyDir:
            medium: Memory
            sizeLimit: 2Gi
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-qwen3-8b
spec:
  selector:
    app: vllm-qwen3-8b
  ports:
    - port: 8000
      targetPort: 8000

The image starts vllm serve, so args holds only the model and the flags. The startup probe allows 90 checks 10 seconds apart, which is 15 minutes. Our DGX Spark took about 8 minutes for its first start, so set the limit from your own measured cold start. The probes stop Kubernetes from sending traffic or restarting the pod while the weights load.

Replace the emptyDir cache with a PersistentVolumeClaim, so a new pod does not download the weights again. The memory volume at /dev/shm gives the processes shared memory, as --ipc=host does with Docker. The API key does not protect /health or /metrics, so also limit which pods can reach the Service with a NetworkPolicy.

This is enough for a few replicas of one model. A plain Service spreads requests at random, so a user's next turn can go to a replica that never cached the earlier turns. An autoscaler on CPU use also scales on the wrong signal, because GPU load is not CPU load.

Pick a layer for the features you need

llm-d published its version 1.0 on 2026-10-20. It joined the CNCF as a sandbox project in March 2026. Its router follows the Gateway API Inference Extension, a Kubernetes project for routing to model servers.

Know which vLLM features the layers use

The layers read the same vLLM interfaces. Check that your vLLM version supports the ones your layer needs:

• Metrics. The routers read vllm:num_requests_waiting, vllm:num_requests_running, and vllm:kv_cache_usage_perc to pick a replica.
• KV cache events. With --kv-events-config, vLLM publishes each block it stores and removes. A router uses these events to send a request to the replica that holds its prefix.
• KV connectors. The prefill and decode split and the shared cache use --kv-transfer-config.
• Health. /health serves the startup, liveness, and readiness probes.

Not measured on our hardware. We have no cluster run, and we did not apply the manifest above.

Your hardware is not an NVIDIA GPU. When you finish this section, you can find the plugin for your hardware and its version.

Know what is built in and what is a plugin

vLLM 0.31.0 has built-in support for NVIDIA GPUs, AMD GPUs through ROCm, Intel GPUs, and CPUs. Other hardware runs through a plugin. A plugin is a separate Python package that registers a platform with vLLM when it is installed. Install it next to vLLM, and vLLM finds it.

The plugin design came from an RFC in December 2024. Since then, several backends that lived inside vLLM moved out into plugins.

Find your plugin

Check the version before you plan

A plugin often supports an older vLLM than the newest release. In October 2026, the newest final release of vllm-ascend targeted vLLM 0.23.0, while vLLM itself was at 0.32.0. The TPU plugin had a release for vLLM 0.31.0 within days of it.

Before you plan a feature on your hardware:

1. Find the newest plugin release, and read the vLLM version it targets.
2. Read that vLLM version's documentation, not the newest one.
3. Check the feature table of the plugin. Some features of the CUDA path, such as some quantization formats and attention backends, have no version on other hardware.

Not measured on our hardware. We ran only NVIDIA GPUs.

Something is wrong and you know what you see, not why. When you finish this section, you can match the symptom to a cause, run one check, and apply one fix.

Find your symptom in the list. Each entry gives the likely cause, the check that confirms it, and the fix. Issue numbers refer to the vLLM repository, with their state at version 0.31.0. The entries marked "from the vLLM documentation" come from the vLLM troubleshooting guide at 0.31.0, and we did not reproduce them.

The server does not start

The start hangs while it downloads the model

• Cause: the download is slow, or the model needs a Hugging Face token. From the vLLM documentation.
• Check: the log stops before the Model loading took line, and the Hugging Face cache folder grows slowly or not at all.
• Fix: download the model first with the hf download command, and set HF_TOKEN for a model with restricted access. Then set HF_HUB_OFFLINE=1, so vLLM uses the local copy only.

The start stops with a message that the model is not supported

• Cause: the model architecture is not in vLLM 0.31.0, or a missing package stops vLLM from importing the model code. From the vLLM documentation.
• Check: the error names the architectures that failed, such as Model architectures [...] are not supported for now or failed to be inspected.
• Fix: check the supported models list for your version. If the model is in the list, read the log above the error for an import failure.

The start stops with a message about the driver or the PTX toolchain

• Cause: the CUDA build of vLLM is newer than your NVIDIA driver supports. From the vLLM documentation.
• Check: the error says The NVIDIA driver on your system is too old, or that the PTX was compiled with an unsupported toolchain.
• Fix: update the driver. In the official image, on some datacenter GPUs, -e VLLM_ENABLE_CUDA_COMPATIBILITY=1 turns on the CUDA compatibility libraries.

The start stops with a message about the KV cache and the maximum length

• Cause: one request at --max-model-len needs more KV cache than is left after the weights.
• Check: the error gives the largest length that fits.
• Fix: set --max-model-len to your longest request. The startup page has the details.

The start stops with a message about free memory and the memory fraction

• Cause: another process holds GPU memory, so the share you asked for is not free.
• Check: run nvidia-smi and look at the processes on the GPU.
• Fix: lower --gpu-memory-utilization below the free share, or stop the other process.

The server crashes at start after you set an FP8 KV cache

• Cause: the FP8 cache moved vLLM to the FlashInfer backend, and FlashInfer could not compile its kernels. Issue 60262 tracks this, and it is open.
• Check: find the line that names the attention backend, and the FlashInfer error after it.
• Fix: add --attention-backend TRITON_ATTN. The FP8 KV cache page shows our case.

The first start stops with a message that nvcc is missing

• Cause: FlashInfer compiles its kernels at the first start, and the host has a driver but no CUDA toolkit. We saw this on a new uv install of 0.31.0 on our L4 node.
• Check: the error comes from flashinfer/jit/cpp_ext.py and says Could not find nvcc and default cuda_home='/usr/local/cuda' doesn't exist.
• Fix: install a CUDA toolkit, set CUDA_HOME to it, and start again. Or use the official image, which has the toolkit.

The sampler fails to compile after you set CUDA_HOME

• Cause: the toolkit on the host does not match the CUDA build of PyTorch, so the FlashInfer sampler cannot compile. With a host toolkit at nvcc 13.4, we saw this on our L4 node.
• Check: the error says CUDA compiler and CUDA toolkit headers are incompatible.
• Fix: install the toolkit version that matches your PyTorch build. Or start with VLLM_USE_FLASHINFER_SAMPLER=0, as the FP8 KV cache page does.

A DGX Spark runs out of memory although the fraction leaves room

• Cause: on unified memory, --gpu-memory-utilization does not always limit what vLLM takes. Issue 46307 reports about 112 GiB in use at a fraction of 0.70, and it was closed as stale with no fix. Issue 44740 reports a KV cache that was too large with MTP speculation on the GB10, and it is open.
• Check: compare the Available KV cache memory line with the share you expect, and watch free -g during the start.
• Fix: set the cache size in bytes with --kv-cache-memory-bytes. The DGX Spark page has the details.

The first start on a DGX Spark takes many minutes

• Cause: the weight load copies each tensor from the file map, and that copy is slow on the GB10. Issue 58726 tracks this, and it is open.
• Check: read the Model loading took line. Our 57 GiB load took 330 seconds.
• Fix: keep the weights on a local drive and keep the server running. The DGX Spark page has the details.

The start fails in the compile step

• Cause: a broken compile cache, or a Triton library that does not match your PyTorch. From the vLLM documentation.
• Check: the error comes from torch.compile or from the torch/_inductor folder.
• Fix: start once with VLLM_DISABLE_COMPILE_CACHE=1. If that works, delete ~/.cache/vllm/torch_compile_cache. Use -O0 to start without the compile while you debug.

A server on several GPUs hangs at start

• Cause: the GPUs cannot reach each other through NCCL, because of the network, the drivers, or a missing shared memory mount. From the vLLM documentation.
• Check: start again with NCCL_DEBUG=TRACE and read where the log stops. Run the NCCL test that the parallelism page describes.
• Fix: give a container --ipc=host or a large /dev/shm. On PCIe machines, the vLLM guide suggests NCCL_CUMEM_HOST_ENABLE=0 or a driver update for a peer access error.

The maximum length is shorter after an upgrade

• Cause: release 0.30.0 changed how vLLM reads context scaling such as YaRN, and some models now get a shorter derived length.
• Check: compare the derived length in the startup log with the old one.
• Fix: set --max-model-len yourself, after you confirm that the model supports the length.

Requests fail while the server runs

A request returns HTTP 400 about the maximum context length

• Cause: the prompt tokens plus max_tokens are more than --max-model-len.
• Check: the error says This model's maximum context length is and gives the counts. When max_tokens alone is above the limit, the error says that max_tokens cannot be greater than max_model_len, as it did on our L4.
• Fix: send a shorter prompt or a smaller max_tokens, or start the server with a larger maximum length if the pool allows it.

Every request fails after an error in the engine

• Cause: the engine core process stopped, for example after a GPU error. The API server stays up, but it cannot serve. From the vLLM source.
• Check: the log shows EngineCore encountered an issue, and the real error is in the lines above it.
• Fix: fix the cause in the earlier lines, then restart the server. Let your orchestrator restart it, because the server does not recover alone.

The server is slow

The first token takes seconds, but the stream is fast after it

• Cause: requests wait in the queue because the pool or the batch is full.
• Check: vllm:num_requests_waiting is above 0 and vllm:kv_cache_usage_perc is near 1.
• Fix: serve fewer users on the GPU, or make room in the pool with an FP8 KV cache or a shorter maximum length.

The stream has long gaps when many users are active

• Cause: new prompts share decode steps with running requests, and long prompts take a large part of each step.
• Check: compare the P99 ITL with the median ITL. A large gap means stalls, not a slow GPU.
• Fix: lower the token budget, as the token budget page shows. On our machines the gaps also depended on the prompt lengths and on the engine, as the engine page shows.

The preemption counter rises

• Cause: a running request needed a new block and none was free. vLLM stops the request and computes it again later.
• Check: vllm:num_preemptions_total rises, and the periodic log line shows a preemption count.
• Fix: raise --gpu-memory-utilization, or lower --max-num-seqs or --max-num-batched-tokens.

N-gram speculation gives less than you expected

• Cause: at 0.31.0, the n-gram method makes vLLM use the older model runner, and it turns off async scheduling.
• Check: the startup log has a warning that async scheduling is off.
• Fix: compare it with no speculation on your traffic, with the same model runner on both sides.

The first requests after a start in eager mode are slow

• Cause: since 0.31.0, --enforce-eager also turns off the warmup that compiles kernels at startup, so the first requests compile them.
• Check: the first few requests are slower than the rest.
• Fix: send a few warmup requests before you serve traffic, or use -O0 instead.

NVFP4 decode is slower on a DGX Spark than on older releases

• Cause: issue 59770 reports a slower decode for one NVFP4 model on the GB10 since 0.29.0. It is open.
• Check: compare the time per token on your version and on 0.28.
• Fix: pin the faster release for that model until the issue closes.

The output is wrong

The answer starts with a long thinking block

• Cause: a reasoning model, such as Qwen3, runs without a reasoning parser.
• Check: the content field starts with <think>.
• Fix: start the server with the parser for the model, such as --reasoning-parser qwen3. The reasoning models page has the details.

The model repeats one token after a prefix-cache hit

• Cause: issue 60174 reports corrupt output on 0.30 and 0.31 when DFlash2 or DSpark speculation runs with the prefix cache. From the third request, each answer that starts from cached blocks is one token, over and over. It was closed on 2026-10-22, after 0.31.0, so check that your version has the fix.
• Check: the problem starts only after a cache hit, and it goes away with --no-enable-prefix-caching.
• Fix: turn off the prefix cache or the speculation method for that model.

JSON output is invalid with speculative decoding on

• Cause: a JSON schema and a speculation method can disagree on the next tokens. Issue 60830 showed this for MTP on one model, and it is closed.
• Check: run the same requests without --speculative-config.
• Fix: upgrade, or turn off speculation for JSON traffic.

The output changes from run to run

• Cause: vLLM does not promise the same output by default. The batch that a request joins changes the order of the sums. A prefix-cache hit can also change a greedy output. In our runs with Qwen3-0.6B at one user, a warm cache changed 11 of 32 greedy outputs. That GPU was a shared L4 with no clock lock. A change of dtype, such as BF16 to FP16, also changes most outputs.
• Check: send the same request alone twice. If the answers match, the batch causes the change. If they differ, the second request can hit the prefix cache. Repeat the test with --no-enable-prefix-caching. If the answers now match, the cache causes the change.
• Fix: offline, set VLLM_ENABLE_V1_MULTIPROCESSING=0. Online, the batch-invariant mode helps, but at 0.31.0 it does not cover every path. Reset the prefix cache between runs that you compare. The reproducibility page covers it.

The benchmark is wrong

The second benchmark run is faster than the first

• Cause: with a fixed seed, the random dataset sends the same prompts again, and the prefix cache serves them. In our test with Qwen3-0.6B at 16 users, a repeat with no reset read a median TTFT about 6 times lower. That GPU was a shared L4 with no clock lock.
• Check: vllm:prefix_cache_hits_total rises during the second run.
• Fix: reset the prefix cache before each run, use a new seed, and record the cached tokens. The benchmark page gives the protocol. The POST /reset_prefix_cache route exists only with VLLM_SERVER_DEV_MODE=1, so use it only on a server that is not exposed.

Your server must survive restarts and upgrades. When you finish this section, you can deploy a server that starts fast and upgrades safely.

Pin the image

Pin an exact version, such as vllm/vllm-openai:v0.31.0, never latest. A new minor release comes about every two weeks, and each one changes defaults or removes flags. Between 0.30 and 0.32, for example:

• Release 0.31 made --enforce-eager also skip the kernel warmup.
• Release 0.31 renamed a Mamba prefix-cache flag and removed the slow tokenizer mode.
• Release 0.32 dropped Python 3.10 and changed the default CORS origins from every origin to none.

The last change breaks any browser client that calls the server directly. After an upgrade to 0.32, set --allowed-origins to the origins you trust.

Plan for the cold start

A start has three slow parts: the weight load, the compile, and the CUDA graph capture. On our DGX Spark, the model load alone took 330 seconds for 57 GiB.

• Weights. Keep them on a local disk. A network mount or a fresh download makes every start slower.
• Compile. vLLM caches compiled code under ~/.cache/vllm. Keep that folder on a persistent volume, so a restart reuses it.
• Preload. Release 0.31 added vllm preload, which loads and warms a model before the server needs it.

vllm bench startup measures a cold and a warm start. Run it once for each new image.

Not measured on our hardware. We have not timed a warm start with a kept compile cache.

Check health before traffic

Use /health for the liveness probe. For readiness, send one short real request, which proves that the engine can generate. Give both probes a start delay longer than your measured cold start.

Shut down cleanly

Send SIGTERM to the server process, not a forced kill. Before the shutdown, remove the pod from the load balancer so that no new requests arrive. Give the pod a grace period at least as long as your longest request. A forced kill drops every running request.

Change settings in this order

Change one setting at a time, and measure after each change:

1. The memory: the maximum length and the memory fraction.
2. The concurrency: --max-num-seqs and the token budget.
3. The data types: an FP8 KV cache, then FP8 weights.
4. The speed features: speculative decoding and the compile level.

Each later step depends on the ones before it. A speed feature measured on a server that runs out of memory measures the memory, not the feature.

Read the security advisories

The vLLM project published dozens of security advisories in the weeks around 0.31 and 0.32. Several were rated high. Many came through multimodal inputs, and some through the KV transfer connectors. Do not expose the server to the internet without a gateway in front of it. Use --api-key, as the server options page shows, and read the advisories before you pick a release.

vllm serve with one GPU starts three processes, and the count grows with every GPU you add. When you finish this section, you can name each process, say what it owns, and predict the process count for a parallel setup.

Three jobs that compete for one CPU

A serving engine does three kinds of work. It speaks HTTP and turns text into tokens and back. It decides which requests run in the next step. It runs the model on the GPU.

In one Python process, these jobs take turns on one interpreter lock. While the server tokenizes a long prompt, the scheduler waits, and the GPU waits for the scheduler. On a fast GPU, a decode step takes only milliseconds. A few milliseconds of Python work between steps then leaves the GPU idle for a large share of the time.

vLLM gives each job its own process:

The API server and the engine core exchange messages over ZeroMQ sockets. The engine core and its workers share memory queues. A data-parallel setup adds one more process, a coordinator that keeps the ranks in step.

Count the processes

The vLLM design documents give a formula for the count. Add the API servers, the data-parallel engine cores, and the GPU workers, plus one coordinator for data parallelism. A server with -tp 4 runs one API server, one engine core, and four workers, so six processes.

This matters when something goes wrong. A crash in a worker shows up in the engine core log. A slow tokenizer shows up as a long time to first token while the GPU is idle. When you read a log, first find which process wrote the line.

Why not threads

Threads in one Python process share the interpreter lock, so they would not remove the wait. Separate processes also let vLLM restart or replace one part. The repository also has an experimental front end written in Rust. It replaces the Python API server and talks to the same engine core over the same sockets.

Separate processes have a cost: each request and each output is copied across a process boundary. The copy is small next to a model step, so the design accepts it. An environment variable, VLLM_ENABLE_V1_MULTIPROCESSING=0, runs the engine core in the API server process instead. The reproducibility guide uses it for offline runs, because one process makes the order of scheduling repeatable.

Most explanations of LLM serving describe a prefill phase and a decode phase. The vLLM scheduler has neither. When you finish this section, you can explain what the scheduler does in one step, and why that design makes chunked prefill the default.

One question for every request

The scheduler asks the same question about every request: how many tokens does this request still need to compute? A new request needs all of its prompt tokens. A running request needs one token, or more with speculative decoding. A comment in the scheduler source says the same in its own words. No decode phase and no prefill phase exist, only a count of tokens still to compute.

In each step, the scheduler has a budget of tokens, set by --max-num-batched-tokens. For vllm serve on a GPU with less than 70 GiB, such as the L4, the default is 2,048. It spends the budget in this order:

1. It gives each running request the tokens it needs, up to the budget.
2. It admits waiting requests while budget and room for running requests remain. --max-num-seqs limits the running requests. For vllm serve, the default is 256 on a GPU with less than 70 GiB and 1,024 on a larger one other than an A100.

A long prompt that does not fit the rest of the budget gets a part of the budget now and the rest in later steps. That is chunked prefill. It is on by default, because the budget design needs nothing else to support it.

Why decode goes first

Running requests come first, so a user who already sees a stream keeps seeing it. A new prompt waits for the budget that is left. The vLLM documentation states the trade. A smaller budget gives a smoother stream, and a larger one gives a faster first token for long prompts. The small default of 2,048 is a choice for the stream. GPUs with 70 GiB or more get 8,192, and the long prompts page lists the rule.

What happens when memory runs out

A running request needs a new KV block every 16 tokens. If no block is free, the scheduler preempts a request. Under the default first-come policy, it picks the newest running request. It frees all of its blocks and puts the request back at the head of the waiting queue. When the request runs again, vLLM computes its tokens again from the start.

Older vLLM versions could also copy the blocks to CPU memory and back. The current engine only recomputes, because the documentation judges the recompute cheaper in this architecture. The cost shows up as a longer time for the preempted request, and as the vllm:num_preemptions counter.

Why one budget

One budget for every kind of work keeps the scheduler simple. Prefix caching only lowers the count of tokens to compute. Speculative decoding only raises the count for a running request. The same loop serves all of them. The cost is that one number decides the balance between the first token and the stream. No single value of that number suits every workload.

vLLM started as a paper about one idea: store the KV cache in fixed-size blocks, the way an operating system stores memory in pages. When you finish this section, you can explain how vLLM finds a block again for a repeated prefix. You can also say which block leaves first.

Blocks, not one buffer for each request

At startup, vLLM splits the KV cache into blocks of 16 tokens and keeps all of them in one pool. A request gets blocks one at a time as it grows. Its blocks do not need to be next to each other in memory. A small table maps the request's token positions to block numbers.

This removes the two wastes of a buffer for each request. No memory sits reserved for a length the request never reaches. No gaps form between requests of different lengths. The original paper called this PagedAttention.

Finding a prefix again

When a block fills, vLLM computes a hash of its tokens together with the hash of the block before it. The chain means that one hash names a whole prefix, not only 16 tokens. The hash also covers other inputs that change the keys and values. These are the LoRA adapter, the images in the prompt, and an optional salt for each request.

A new request computes the same chain of hashes over its prompt. The engine looks each hash up in a map and stops at the first miss. Every block before the miss is a hit, and its tokens need no compute. Only full blocks are cached, so the last part of a prompt below 16 tokens is always computed.

Since version 0.11, the hash is SHA-256 by default. A faster hash exists as an option, but the documentation warns that a collision could return the wrong cache entries, or leak data between tenants.

Eviction

A block that no request uses goes to a free list, and it keeps its hash. A later request with the same prefix can still use it. When vLLM needs a new block, it takes the block at the head of the free list, which is the one unused the longest. Only then does the old content go away.

So the cache is the pool. No separate cache memory exists, and a busy server keeps fewer old prefixes than an idle one.

Hybrid models get groups

A model with sliding-window layers or state-space layers does not keep one entry for each token in every layer. The engine puts each kind of layer in its own group. Each group has its own page size and its own rule for a cache hit. A sliding-window layer, for example, needs only the blocks inside the last window. The groups share one pool. The long-context page covers what this means for your settings.

The attention kernel is the one part of the forward pass that changes most between GPUs. When you finish this section, you can predict the backend vLLM picks on your GPU, and explain why one flag change can swap it.

A list for each GPU class

vLLM 0.31.0 has more than 30 attention backends. On a CUDA GPU, it walks an ordered list and takes the first backend that can run the model with your settings. For a model with standard attention, the list is:

Models with multi-head latent attention, such as the DeepSeek family, have their own lists.

FlashAttention itself has versions. vLLM uses version 3 on Hopper GPUs and version 4 on the Blackwell datacenter GPUs where it is supported. Every other GPU gets version 2, so both our L4 and our GB10 ran FlashAttention 2. Our GB10 log names the choice and the list:

(EngineCore pid=143) INFO 10-21 10:53:17 [cuda.py:526] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].

Why a flag changes the backend

Each backend declares what it supports: the data type of the KV cache, the head size, the block size, and the CUDA graph modes. If your settings ask for something the first backend cannot do, vLLM moves to the next one.

The FP8 KV cache is the common case. On an L4, FlashAttention 2 cannot read an FP8 cache. With --kv-cache-dtype fp8, vLLM skips it and picks FlashInfer if it is installed, or Triton if it is not. Your FP8 result then measures a new data type and a new kernel at the same time. On our L4 the FlashInfer build failed, as the FP8 KV cache page shows.

Why the order differs

Our reading: the order reflects which kernels were fastest and best tested on each class when the list was written. FlashInfer has kernels tuned for the Blackwell datacenter GPUs, so it comes first there. On older classes, FlashAttention has the longer record. The list is a judgment that changes between releases, not a law.

That is why the manual tells you to read the backend line after every start, and after every flag change. If two runs used different backends, you compared kernels, not settings. --attention-backend pins one backend by its enum name, such as TRITON_ATTN.

At 0.31.0, vLLM has two model runners, and the one you get depends on your flags. When you finish this section, you can say which runner your server uses, and why a speculation method can change it.

What a model runner does

The model runner lives in each GPU worker. It takes the batch that the scheduler chose, builds the input tensors, runs the forward pass, and samples the next tokens. It also loads the weights, runs the startup profile, and captures the CUDA graphs.

The first runner of the V1 engine grew to more than 7,500 lines. The design notes of the vLLM team say that the first version had design mistakes and technical debt. The team then built Model Runner V2 from first principles. The new runner is about a third of that size.

Two ideas

A fixed table of request rows. Model Runner V2 keeps a persistent batch with one row for each --max-num-seqs slot, 256 on our GPUs. Each request owns one row for its whole life. A preempted request leaves its row, the same as a finished one. The runner updates only what changed in each step, instead of building the batch again.

No waits for the CPU. The runner keeps its work on one CUDA stream and avoids points where the CPU must wait for the GPU. That lets the scheduler plan the next step while the GPU still runs the current one. The name for this is async scheduling, and it is on by default for most generate models.

When you get the old runner

Model Runner V2 is the default on CUDA when Triton is installed. vLLM 0.31.0 uses the old runner instead in these cases, among others:

• The n-gram, suffix, Medusa, or MLP speculator methods.
• The Mamba cache mode all, which release 0.32 removed.
• The stock torch.compile mode, and some parallel features such as sequence parallelism with tensor parallelism.

VLLM_USE_V2_MODEL_RUNNER=0 or =1 pins the choice.

This matters most for speculation. EAGLE3 keeps the new runner and async scheduling. N-gram speculation moves you to the old runner and turns async scheduling off. A comparison of the two methods is then also a comparison of two runners.

Where it goes next

The vLLM roadmap for the fourth quarter of 2026 lists the full removal of the old runner. Each release since 0.30 moved more features to the new one. If a flag puts you on the old runner, check the next release before you build on that combination.

vLLM spends the last part of every start on compiling and capturing. When you finish this section, you can say what each optimization level turns on, and why a full CUDA graph cannot cover every batch.

Two separate tools

torch.compile traces the model's Python code into a graph and generates fused kernels. Fusion joins small operations, such as a norm and a quantize step, into one kernel. The compiled code goes to a cache under ~/.cache/vllm/torch_compile_cache. A second start with the same settings skips most of the work. The server finishes all compiles before it serves the first request.

A CUDA graph records a sequence of kernel launches with fixed shapes and replays it with one CPU call. A decode step for a small batch is hundreds of short kernels, and a graph removes the CPU cost of launching each one.

The graph modes

A graph needs fixed shapes, and attention does not always have them. A batch can mix prompts of different lengths. vLLM handles this with modes:

The default is FULL_AND_PIECEWISE. At run time, a dispatcher picks a mode for each batch from its token count, its request count, and whether it is uniform. If the attention backend cannot replay a full graph for a batch, the dispatcher uses the piecewise one.

On the L4 and the GB10, FlashAttention 2 supports full graphs only for uniform batches, such as pure decode steps. A mixed batch with a new prompt runs piecewise.

The optimization levels

-O0 to -O3 set compile and graphs together:

A flag that you set yourself wins over the level. --enforce-eager turns off compile and graphs in one switch. Since 0.31.0 it also turns off the kernel warmup at startup, which -O0 keeps.

Capture sizes

vLLM captures graphs for a list of batch sizes. A batch larger than the largest size runs without a full graph. --max-cudagraph-capture-size sets the largest size, and --cudagraph-capture-sizes sets the list. More sizes cost startup time and GPU memory, which comes out of the KV pool.

vllm bench startup measures the start with a cold and a warm compile cache, so you can see what each choice costs.

Speculative decoding promises more tokens for each step without a change to the output. When you finish this section, you can explain how vLLM keeps that promise, and why the gain shrinks as the batch grows.

Guess, then check

A decode step on a small batch reads all the weights to produce one token for each request. Most of the GPU sits idle while it waits for memory. Speculative decoding uses that idle time.

A cheap proposer guesses the next few tokens. The target model then runs once over the guesses and computes its own distribution at every position. A rejection sampler walks the guesses in order. It keeps each guess that the target model's distribution supports, and at the first rejection it samples a correct token from the target and stops. One step therefore gives at least one token and at most the number of guesses plus one.

The check is what keeps the output the same. With greedy sampling, a guess stays only if it equals the target's top token. With random sampling, the rejection rule keeps the output distribution equal to the target's distribution.

The proposers

vLLM 0.31.0 has several kinds of proposer, set with --speculative-config:

N-gram and suffix methods cost almost nothing, and they help when the answer copies the prompt, as in code edits. A draft head like EAGLE3 needs a trained head for your model, and it helps on open text. For Qwen/Qwen3-8B, a public EAGLE3 head exists with one layer and a draft vocabulary of 32,000 tokens.

Why the gain falls with load

The idle time that speculation uses exists only while the step reads more memory than it computes. As the batch grows, the step does more compute for each byte, and the idle time goes away. The checks of rejected guesses then cost real compute. The vLLM documentation rates the gain of EAGLE as high at a low request rate and lower at a high one.

The dynamic mode follows from this. It sets the number of guesses from the batch size and stops guessing for large batches. At 0.31.0, its online example runs on the older model runner.

What to watch

The engine exports the proposed and the accepted tokens. The acceptance rate and the mean number of tokens for each step tell you whether the proposer guesses well on your text. A high acceptance rate with no gain in the time per token means the batch is already compute-bound.

Some setups need the KV cache of a request in a place other than the GPU that computed it. When you finish this section, you can name the jobs a KV connector does, and say what disaggregated prefill buys and what it does not.

Why move the cache

Three setups need it:

• Disaggregated prefill. One server computes the prompt, and another server generates the answer. The KV cache must move from the first to the second.
• Offloading. Blocks that do not fit on the GPU go to CPU memory or to storage. They come back when a later request needs the same prefix.
• A shared cache. Several servers read prefixes from one store, so a prefix computed once serves all of them.

One interface, many transports

vLLM puts these jobs behind one interface, the KV connector. The scheduler asks the connector which blocks of a request it can load from outside. It then counts those tokens as computed, the same as a prefix-cache hit. The worker asks the connector to load or save the blocks around each forward pass.

At 0.31.0, the repository has connectors for several transports:

Set the connector with --kv-transfer-config, which gives the connector name, the role of the server, and the options of the transport. The role says whether the server produces the cache, consumes it, or does both.

What disaggregation buys

The vLLM documentation says that disaggregated prefill does not raise throughput. It gives two other reasons to use it. You can tune the first token and the stream on separate servers. You can also keep long prompts from stalling the stream of other users, which controls the tail of the inter-token latency.

The price is a transfer for every request, and more machines to run. If a load fails, the default policy is to fail the request. A recompute policy exists as an option.

Where it goes

The connector interface is one of the most active parts of the code. In October 2026, the open work had three parts. One was KV offloading that does not depend on the tensor-parallel size. Another was a compression layer for the transferred blocks. The last was a fix for blocks released during a remote read. Pin the version when you build on a connector, and read its open issues first.

The vLLM you install today is the second engine under the same name. When you finish this section, you can say where the project started, why it rewrote its engine, and who decides its direction now.

A Berkeley project with one idea

The repository started on 2023-02-09 at the University of California, Berkeley, with commits by Woosuk Kwon. The first public release, v0.1.0, came on 2023-06-20, together with a blog post that introduced PagedAttention. The paper on the same idea appeared at SOSP 2023.

The idea was to store the KV cache in fixed blocks, the way an operating system stores memory in pages. A server could then fit more requests on one GPU, and requests could share blocks. Early users ran it behind the Chatbot Arena and the Vicuna demos of the LMSYS team.

Features in the first engine

Between 2023 and 2025, the first engine gained most of the features this manual covers:

Our reading: each feature went into a design that did not plan for it, and the code grew harder to change.

The rewrite

The team built a second engine, called V1. Its goals were a simple and modular code base, almost no CPU overhead, and the main optimizations on by default. V1 appeared for testing in v0.7.0 on 2025-01-27, with torch.compile on by default. It became the default engine in v0.8.0 on 2025-03-18. Release v0.11.0 on 2025-10-02 removed the old engine.

The ideas on the other pages of Part 8 come from this rewrite. One token budget replaced the prefill and decode paths. Chunked prefill and prefix caching became the default. The process split put the scheduler in its own process. Model Runner V2 later rebuilt the model runner, which the team said had design mistakes.

Releases now

Until v0.11, a new minor release came every one to five months. Since v0.12.0 in December 2025, a new minor release comes about every two weeks. By 2026-10-18 the project had published 108 releases on GitHub. Each release changes defaults and removes deprecated flags, so a recipe from a few months ago can be wrong today. This manual pins 0.31.0 for that reason.

Who runs it

The project started the incubation process at the LF AI and Data Foundation in July 2024. On 2025-05-07, it became a hosted project of the PyTorch Foundation. Five lead maintainers and 55 active committers review changes. Companies contribute engineers to the project. Red Hat brought in the Neural Magic team when it bought that company in January 2025.

The pace has a cost that the project discusses in public. In September and October 2026, two RFCs on the project said that review cannot keep up with the number of pull requests. One proposed a reviewer role. The other proposed evals and benchmarks that contributors run on their own GPUs, so that a pull request comes with its evidence.

Use this page to look up a flag that a section names.

Defaults are for vllm serve at 0.31.0 on a CUDA GPU, read from the configuration classes in the source.

Server

Memory and length

Scheduling

Cache

Execution

Parallelism

Environment variables

Use this page to find a metric that a section names.

vLLM exports the metrics at /metrics in the Prometheus format. Every name starts with vllm:.

Every counter shows on /metrics with a _total suffix, as these tables list it.

Load and memory

Latency

Cache and tokens

Speculative decoding

What our runs recorded

Our benchmark client recorded TTFT, TPOT, ITL, and throughput for each request. A sampler recorded the GPU clock and temperature every second. We did not scrape /metrics during the sweeps.

Use this page to check whether a problem you see is known.

States are as of 2026-10-22. Numbers refer to issues in the vLLM repository.

Issues

Our own incidents

Use this page to match your setup to ours before you compare a number.

vLLM

The machines

Models

The comparison engine and the client

Use this page when a section uses a term you do not know.

Use this page to judge how much weight a statement in the manual carries.

When two sources disagree, the higher one wins, and the page says so.

What we did not use

We did not use benchmark numbers from vendors or from other teams as evidence for a decision. Where the manual reports a result from another team, it names the team once and says that the number is theirs.

Use this page to check a number from another section, or to repeat a run.

Method

• Hardware and versions. The version pins page lists them.
• Client. Every run used the SGLang 0.5.22 benchmark client. The vLLM runs used --backend vllm, which sends each request to /v1/completions. The client sends one warmup request first, and it asks the server to ignore the end-of-sequence token unless you pass --disable-ignore-eos. Its random dataset cuts or repeats real chat text to the asked length. The benchmark page prints the full command.
• Waves. The waves workload sends 64 prompts at 16 users. The length ratio sets how far the prompt and output lengths can vary. At 1.0 every request has 512 prompt and 128 output tokens. At 0 each length is drawn at random from 1 token up to that size.
• Seeds. The runs to 2026-10-21 used seeds 1, 2, and 3. Most runs on 2026-10-22 used seeds 1 and 2, and the tables give the count. Tables give the median, and some columns give the lowest and the highest.
• Server settings. The sweep tool starts every vLLM server with VLLM_USE_FLASHINFER_SAMPLER=0, because the FlashInfer sampler fails to build on our L4. It also sets --seed 0, --max-model-len, --num-gpu-blocks-override, and --gpu-memory-utilization. The L4 used 8,192 tokens, 1,506 blocks, and 0.9. The GB10 used 8,192 tokens, 14,685 blocks, and 0.75. The shared L4 GPU used 2,048 or 2,304 tokens, 183 blocks, and 0.83. The small-pool run used 4,096 tokens and 410 blocks.
• Rounding. The tables and the sections use one rule. They round the decimal value from the summary file, and a half goes up, so 1,012.5 shows as 1,013.
• Pools. On the L4, vLLM ran with --num-gpu-blocks-override 1506, a pool of 24,096 tokens, against 24,094 for SGLang. On the GB10, vLLM ran with 14,685 blocks, a pool of 234,960 tokens. The SGLang pools on the GB10 were 233,303 tokens for load, 234,958 for waves, and 251,910 for mixed.
• Clocks. A sampler recorded the SM clock and the temperature every second. The sweep tool can wait for the GPU to cool before a run. It can also repeat a run when the median clock falls below a clock floor, a lowest clock that we set for each sweep. The clock column shows the result for each point. On 2026-10-22, most runs had the SM clock locked at 900 MHz with nvidia-smi -lgc. A lock sets the highest clock, but a GPU at a high temperature still slows below it. The sections compare two points only at similar clocks, and they say so where a point ran below the lock or in a throttled session.
• Dates. The L4 load runs of vllm and sglang were on 2026-10-17, and the vllm-b, vllm-async, vllm-eager, and vllm-seqs-16 load runs on 2026-10-21. The L4 rate and waves runs were on 2026-10-18. The L4 prefix runs with the cache on ran from 2026-10-17 into 2026-10-18, and the run without the prefix cache was on 2026-10-21. Every GB10 run in these tables was on 2026-10-21. Every run in the three L4 tables for 2026-10-22 was on that day. The runs on the shared GPU of the L4 node had a pool of 2,928 tokens, so they tested 1 and 4 users only.

• The cache between user counts. The load tables before 2026-10-23 kept one server for each config and did not reset the prefix cache between user counts. Each seed sent the same first prompts at every count. Below 32 users these tables read the first token too fast, by 3.6 to 5.3 times for vLLM on our L4. The rerun of 2026-10-23 resets the cache before every run, and the benchmark page compares the two.
• Runs of 2026-10-23. The L4 runs of that day used the first GPU with the clock locked at 900 MHz. Each run had two seeds and a cache reset first. The card ran hot, so many runs fell below the lock. Each evidence file gives the median clock of its rows, and the sections compare only rows at 900 MHz.

Not run yet

These runs are planned and are not in these tables:

• --max-num-batched-tokens at 512, 2048, and 16384 on mixed prompts on the L4. The GB10 has 2048 and 16384.
• -O1 on the L4.
• a backend comparison with one KV data type.

Tables

The config vllm-async passes --async-scheduling. For these models with no speculation, that is the default, so vllm-async is a second run of the baseline setting on both machines.

L4, load: users at once, 512 prompt and 128 output tokens

L4, rate: requests each second, 512 prompt and 128 output tokens

L4, waves: 64 prompts at 16 users, by length ratio (1.0 means equal lengths)

L4, prefix: shared prefix tokens, 4 groups of 16 prompts, 8 users

L4, throttled session: 512 prompt and 128 output tokens, 2026-10-22

The GPU ran far below its full clock in this session. Compare these rows only with each other, at matching clocks.

L4, clock locked at 900 MHz: 512 prompt and 128 output tokens, 2026-10-22

The lock sets the highest clock. A GPU at a high temperature still slows below it, as the 1-user runs of vllm-lock-b and vllm-lock-small-pool show.

L4, shared GPU with a pool of 2,928 tokens: 512 prompt and 128 output tokens, 2026-10-22

GB10, load

GB10, mixed: prompts of 1,024 to 4,096 tokens, 64 to 256 output tokens, by rate