Install vLLM and serve your first model

You have an NVIDIA GPU and a model name, and you want your own OpenAI-compatible endpoint. When you finish this section, you can install vLLM, start a server, send it a request, and read the startup log.

We use vLLM 0.31.0 in every command. vLLM publishes a new minor release about every two weeks, so pin the version in your install command.

Check the GPU first

First, check that the driver sees your GPU. Run this command:

nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv

You see one line for each GPU, with its name and its memory. If the command is not found, install the NVIDIA driver before you continue. vLLM does not install the driver for you.

Check the rest of your machine against this table. It comes from the vLLM 0.31.0 install guide. We did not test the lower limits.

Install vLLM

Pick one of three paths. Use the first path unless you have a reason to pick another.

Path 1, uv. uv creates a clean Python environment and picks the PyTorch build that matches your CUDA driver.

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm==0.31.0 --torch-backend=auto

Path 2, pip. Use a new virtual environment with Python 3.12, then install the same version.

python3.12 -m venv .venv && source .venv/bin/activate
pip install vllm==0.31.0

Path 3, Docker. The official image holds vLLM, PyTorch, and CUDA. We pulled vllm/vllm-openai:v0.31.0 on our arm64 DGX Spark, and it ran.

docker run --rm --gpus all --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:v0.31.0 Qwen/Qwen3-0.6B --reasoning-parser qwen3

The image starts the server for you, so with Docker you skip the next step. Mount your Hugging Face cache, or the container downloads the weights again on every start. Read the log with docker logs on the container.

Check the install with vllm --version. You see 0.31.0.

On the uv and pip paths, the first start compiles FlashInfer kernels, so it needs a CUDA toolkit with nvcc, and CUDA_HOME set to it. The Docker image has one. We installed vLLM 0.31.0 with uv on a host with only a driver, and every start stopped with Could not find nvcc. Install the toolkit, set CUDA_HOME, and start again. The troubleshooting page covers this and a toolkit that does not match.

Start the server

Now, start a small model. Qwen3-0.6B has 0.6 billion parameters, so its BF16 weights need about 1.2 GB.

vllm serve Qwen/Qwen3-0.6B --reasoning-parser qwen3 2>&1 | tee server.log

The server listens on port 8000. The first start downloads the weights, so it takes longer than the next start.

The tee command shows the log in the terminal and also writes it to server.log. Later pages search that file with grep, so keep this name for every start.

Qwen3 is a reasoning model. It writes its thinking first, between <think> tags, and then the answer. --reasoning-parser qwen3 tells vLLM to move the thinking out of the answer text. Without it, the thinking stays at the start of the answer. The reasoning models page explains the parser.

The startup log is long. Six lines tell you most of what you need. The listing shows them from our DGX Spark, where we served the larger Qwen3-30B-A3B. Your lines have the same shape with smaller numbers.

(APIServer pid=1) INFO 10-21 10:53:02 [api_utils.py:347]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.31.0
(EngineCore pid=143) INFO 10-21 10:53:14 [gpu_worker.py:479] Using V2 Model Runner
(EngineCore pid=143) INFO 10-21 10:53:17 [cuda.py:526] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=143) INFO 10-21 10:58:45 [model_runner.py:407] Model loading took 56.93 GiB memory and 330.542793 seconds
(EngineCore pid=143) INFO 10-21 10:59:46 [gpu_worker.py:692] Available KV cache memory: 31.32 GiB
(EngineCore pid=143) INFO 10-21 10:59:46 [kv_cache_utils.py:2464] GPU KV cache size: 234,960 tokens, Maximum concurrency for 8,192 tokens per request: 28.68x

Read the lines in this order:

1. The version. It must match the version you pinned.
2. The model runner. vLLM 0.31.0 uses the V2 model runner on CUDA by default. Some features switch it back to V1, and this line shows it.
3. The attention backend. vLLM picks one for your GPU. A change to the KV cache data type can change this line.
4. Model loading. This line gives the memory that the weights took and the time to load them.
5. The available KV cache memory. This line gives the memory that is left for the KV cache after the weights and the startup profile.
6. The KV cache size. This line gives the tokens that fit in the cache and the number of full-length requests that fit at the same time. Part 2 starts from this line.

The server is ready when the log shows Application startup complete.. On our DGX Spark, that took about eight minutes for 57 GiB of weights. The model loading line gives 330.5 seconds, and 325 of those seconds went to reading the weight files.

Send the first request

Open a second terminal. Send one chat request with curl:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3-0.6B", "messages": [{"role": "user", "content": "Name three prime numbers."}], "max_tokens": 64, "chat_template_kwargs": {"enable_thinking": false}}'

The chat_template_kwargs field turns the thinking off for this one request, so 64 tokens are enough for the answer. The reply is one JSON object. The answer text is in choices[0].message.content, and the token counts are in usage. In the server log, a new line shows the request path and 200 OK.

If you leave the thinking on, the model can spend all 64 tokens on it. The thinking then goes to the reasoning field, content is null, and finish_reason is length. We saw exactly this on an L4. Give a thinking request a much larger max_tokens.

The server speaks the OpenAI API, so the OpenAI Python client works without a change to your code:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
stream = client.chat.completions.create(
    model="Qwen/Qwen3-0.6B",
    messages=[{"role": "user", "content": "Name three prime numbers."}],
    stream=True,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

The tokens print one by one as the server streams them. That is your first streamed token.

When the output is different

• The server stops with an out-of-memory error. Another program uses the GPU, or the model is too large. Read the out-of-memory page.
• The answer starts with a long <think> block. The server runs without a reasoning parser. Start it with --reasoning-parser qwen3, or send "chat_template_kwargs": {"enable_thinking": false} in the request.
• The answer is empty and finish_reason is length. The thinking used all of max_tokens. Raise max_tokens, or turn the thinking off for the request.
• The request returns 404. The model name in the request must match the name in vllm serve. Run curl http://localhost:8000/v1/models to see it.
• The answers use different sampling than you set. By default, vLLM applies the sampling settings from the model repository. Start the server with --generation-config vllm to use the vLLM defaults.
• The start takes many minutes. The first start downloads the weights. On a DGX Spark, weight loading is slow in 0.31.0 for large models.