Manuals / vLLM 101 / Part 2 · Size It
2.2

Fix out of memory at startup

One setting, the GPU memory fraction, decides most startup failures, and a shorter maximum length or a lower concurrency cap fixes most of the rest.

The server stopped before it printed the KV cache size line. When you finish this section, you can read the reason from the last lines of the log and pick the one flag that fixes it.

Know what vLLM claims at startup

vLLM does four things before it serves a request:

  1. It loads the weights.
  2. It runs one profile pass with a full batch to measure the activation memory.
  3. It captures CUDA graphs.
  4. It gives the KV cache the rest of its share of the GPU.

The share is --gpu-memory-utilization. The default is 0.92, so vLLM claims 92 percent of the total GPU memory for itself. The number is a limit for one vLLM instance. It is not a fraction of the memory that is free when the server starts. On unified memory, such as a DGX Spark, it is not always a hard limit, as the DGX Spark page explains.

Find your failure in the log

Read the last 30 lines of the server log. Most startup failures match one of three cases.

What the log saysCauseFix
The KV cache needed for one request at the maximum length is larger than the available KV cache memoryThe weights and the profile leave too little for one full-length requestLower --max-model-len. The error gives the largest length that fits.
The free memory on the device is less than the desired GPU memory utilizationAnother process holds GPU memoryLower --gpu-memory-utilization below the free share, or stop the other process
CUDA out of memory during the profile run or the graph captureThe batch in the profile pass or the graphs need more memory than is leftLower --max-num-seqs or --max-num-batched-tokens, or lower --max-cudagraph-capture-size
Fig. 2.3where a start failsdecision
Where the start fails tells you which setting to change A start can fail at three points. If loading the weights fails, the model is too big for the GPU, so use FP8 weights or more GPUs. If the memory check fails, other programs hold memory, so lower the memory fraction. If the KV check fails, the maximum length needs more cache than is free, so lower the maximum length. THE LOG STOPS AT loading weights before the load line memory check free memory is less KV check max seq len needs THE CAUSE weights too big for this GPU memory is shared other programs hold it cache too small for one full request CHANGE THIS FP8 weights or --tensor-parallel-size lower the fraction --gpu-memory-utilization shorter context --max-model-len
The last line before the start fails tells you which of three settings to change. Read down each column: where the log stops, the cause, and the flag to change.

Fix a shared GPU

On a shared GPU, set the fraction yourself. We ran vLLM 0.31.0 on an L4 that another team also used. Their processes held part of the memory, so we set --gpu-memory-utilization 0.83 and a pool of 2,928 tokens.

The first start still failed, because we also set the maximum length to 2,928 tokens. One request at that length needed slightly more KV cache than was free, and the error said that 2,912 tokens would fit:

the maximum length error on the shared L4 GPUtextFull log
(APIServer pid=3681321) INFO 10-22 00:35:04 [api_utils.py:286] non-default args: {'model_tag': 'Qwen/Qwen3-8B', 'host': '127.0.0.1', 'port': 30100, 'model': 'Qwen/Qwen3-8B', 'max_model_len': 2928, 'gpu_memory_utilization': 0.83, 'num_gpu_blocks_override': 183}
(EngineCore pid=3682071) ERROR 10-22 00:36:18 [core.py:1433] ValueError: To serve at least one request with the model's max seq len (2928), (0.4 GiB KV cache is needed, which is larger than the available KV cache memory (0.4 GiB). Based on the available memory, the estimated maximum model length is 2912. Try increasing `gpu_memory_utilization` (which also controls CPU memory on the CPU backend) or decreasing `max_mod
The first line gives the settings. The second line gives the error and the length that fits. Our capture cut the second line at 420 characters. In vLLM 0.31.0, the rest of the message tells you to raise the memory fraction or to lower the maximum length.

We set --max-model-len 2048, which is enough for our 640-token requests, and the server started. Later we raised it to 2,304 tokens, so that our tests with 2,048-token prefixes fit.

So on a shared GPU, the fraction and the maximum length work together. Lower the maximum length to the longest request you send before you lower the fraction.

Set the KV cache size directly

If you know the KV cache size you want, set it in bytes with --kv-cache-memory-bytes. vLLM then ignores --gpu-memory-utilization for the cache. Use this when two vLLM instances share one GPU and each one needs a known pool.

--num-gpu-blocks-override sets the number of 16-token blocks for a test, as in our fixed pool. Do not use it in production, because it does not check that the memory exists.

Check the fix

Start the server again with tee server.log, as in the first start, and wait for the pool line:

grep "GPU KV cache size" server.log

If the line is there, the server has a pool. Compare its size with your count.

vLLM 0.31.0 · db9527a · 2026-10-05 · Edition 2026.10