Manuals / vLLM 101 / Part 2 · Size It
2.1

Count the memory before you pick a GPU

The KV cache, not the weights, decides how many users fit, and you can count it from four numbers in the model config.

You chose a GPU and the server holds fewer users than you planned. When you finish this section, you can predict the KV pool and the number of users for a model and a context length.

Our result: on our L4 with Qwen3-8B, the 24,096-token pool fit 37 users with 640 tokens each, although the startup line printed 2.94.

Count the bytes for one token

Every token in a running request keeps its keys and values in GPU memory until the request ends. Four numbers in the model config give the size:

KV bytes per token = layers x KV heads x head size x 2 x bytes per value

The factor 2 counts the key and the value. Read the first three numbers from config.json as num_hidden_layers, num_key_value_heads, and head_dim. In BF16, a value takes 2 bytes. With an FP8 KV cache, it takes 1 byte.

ModelLayersKV headsHead sizeKV per token in BF16
Qwen/Qwen3-8B368128144 KiB
Qwen/Qwen3-30B-A3B-Instruct-250748412896 KiB

The mixture-of-experts model has almost four times the weights of the dense model, but it keeps fewer bytes for each token. The KV heads decide this number, not the parameter count.

Fig. 2.1the bytes for one tokenstructure
The bytes one token holds in the KV cache Bytes per token are layers times KV heads times head dimension times two, for keys and values, times two bytes for BF16. Qwen3-8B has 36 layers, 8 KV heads and dimension 128, so 144 KiB. Qwen3-30B-A3B has 48 layers and 4 KV heads, so 96 KiB. LAYERS KV HEADS HEAD DIM K AND V BF16 BYTES MODEL Qwen3-8B 36 x 8 x 128 x 2 x 2 144 KiB for every token of every user Qwen3-30B-A3B 48 x 4 x 128 x 2 x 2 96 KiB for every token of every user
Qwen3-8B holds 144 KiB of KV cache for every token, and Qwen3-30B-A3B holds 96 KiB, because it has fewer KV heads. Multiply the five numbers in each row. The last factor is two bytes for BF16. From the model configs on the Hugging Face Hub.

Turn bytes into users

vLLM gives the KV cache what is left after the weights and the startup profile. Divide that memory by the bytes per token to get the pool in tokens. Then divide the pool by the context that one user holds:

users that fit = KV pool in tokens / (prompt tokens + output tokens per user)

A request that needs more blocks than are free waits in the queue. If a running request needs a new block and none is free, vLLM stops it and computes it again later.

Check the count against the startup log

Read two lines after every start. On our DGX Spark, with Qwen/Qwen3-30B-A3B-Instruct-2507 and --gpu-memory-utilization 0.75, the log said:

the memory lines from a GB10 starttextFull log
(EngineCore pid=143) INFO 10-21 10:58:45 [model_runner.py:407] Model loading took 56.93 GiB memory and 330.542793 seconds
(EngineCore pid=143) INFO 10-21 10:59:46 [gpu_worker.py:692] Available KV cache memory: 31.32 GiB
The weights took 56.93 GiB, and 31.32 GiB was left for the KV cache.

At 96 KiB per token, 31.32 GiB holds about 342,000 tokens.

Fixed pool: in our sweeps, we set the pool size by hand with --num-gpu-blocks-override, which sets the number of 16-token blocks. We gave vLLM the same pool as the SGLang server on the same machine, so that the two engines had the same room. On the DGX Spark the pool was 234,960 tokens, and on the L4 it was 24,096 tokens. Your server sizes the pool from free memory, so it uses the full number.

The next line gives the pool in tokens and the users that fit at the full context length:

the pool line from the L4 comparison runtextFull log
INFO 10-17 20:13:17 [kv_cache_utils.py:2464] GPU KV cache size: 24,096 tokens, Maximum concurrency for 8,192 tokens per request: 2.94x
24,096 tokens at 144 KiB each is 3.31 GiB of KV cache.

The Maximum concurrency number assumes that every user fills the full --max-model-len. Real users hold less, so more of them fit. With 512 prompt tokens and 128 output tokens, the 24,096-token pool holds 37 users at their full length. During our load sweep, the log showed 40 running requests with the cache 99.5 percent full, because users early in their answer hold fewer tokens.

Fig. 2.2the pool decides the userscomparison
The pool in tokens decides how many users fit On the L4 the pool held 24,096 tokens. The startup line reports 2.94 users because it assumes every user fills the full 8,192-token context. With 512 prompt tokens and 128 output tokens, each user holds 640 tokens, so 37 users fit. KV POOL ON THE L4, QWEN3-8B 24,096 tokens, 3.31 GiB at 144 KiB each IF EVERY USER FILLS 8,192 TOKENS Maximum concurrency 2.94x, the number the startup line prints IF EACH USER HOLDS 640 TOKENS 37 users fit, each with 512 prompt and 128 output tokens
The same 24,096-token pool fits 2.94 users at the full context length, but 37 users who each hold 640 tokens. Each blue block is one user. The dashed part is pool that no full-length user fills.

Decide what to change

If the count is too small for your traffic, you have four choices. They are in order of cost:

  1. Lower --max-model-len to the longest context that your users send. The pool stays the same size, and one long request can no longer take all of it.
  2. Use an FP8 KV cache. It halves the bytes per token. The FP8 KV page shows the cost.
  3. Use FP8 weights. They free memory for the cache. The FP8 weights page shows when this pays.
  4. Use a GPU with more memory, or split the model across GPUs. The parallelism page covers this.

vLLM 0.31.0 · db9527a · 2026-10-05 · Edition 2026.10