Manuals / vLLM 101 / Part 8 · How vLLM Works
8.1

Why vLLM runs three kinds of process

This page explains the API server, the engine core, and the workers, and why they are separate processes.

vllm serve with one GPU starts three processes, and the count grows with every GPU you add. When you finish this section, you can name each process, say what it owns, and predict the process count for a parallel setup.

Three jobs that compete for one CPU

A serving engine does three kinds of work. It speaks HTTP and turns text into tokens and back. It decides which requests run in the next step. It runs the model on the GPU.

In one Python process, these jobs take turns on one interpreter lock. While the server tokenizes a long prompt, the scheduler waits, and the GPU waits for the scheduler. On a fast GPU, a decode step takes only milliseconds. A few milliseconds of Python work between steps then leaves the GPU idle for a large share of the time.

vLLM gives each job its own process:

ProcessWhat it ownsHow many
API serverHTTP, the OpenAI routes, tokenization, multimodal input, and the stream back to the clientone by default, or one for each data-parallel rank
Engine corethe scheduler, the KV cache manager, and the loop that sends work to the workersone for each data-parallel rank
GPU workerthe weights and the forward pass on one GPUone for each GPU, so tensor size times pipeline size for each engine core

The API server and the engine core exchange messages over ZeroMQ sockets. The engine core and its workers share memory queues. A data-parallel setup adds one more process, a coordinator that keeps the ranks in step.

Count the processes

The vLLM design documents give a formula for the count. Add the API servers, the data-parallel engine cores, and the GPU workers, plus one coordinator for data parallelism. A server with -tp 4 runs one API server, one engine core, and four workers, so six processes.

This matters when something goes wrong. A crash in a worker shows up in the engine core log. A slow tokenizer shows up as a long time to first token while the GPU is idle. When you read a log, first find which process wrote the line.

Fig. 8.1six processes for four GPUstree
A server with -tp 4 runs six processes vllm serve starts one API server process. It starts one engine core process, which runs the scheduler and the KV cache manager. The engine core starts one GPU worker for each GPU, four workers for tensor parallel size 4. Six processes in total. API server HTTP, tokenizer engine core scheduler, KV manager ZeroMQ worker 0 GPU 0, rank 0 worker 1 GPU 1, rank 1 worker 2 GPU 2, rank 2 worker 3 GPU 3, rank 3 Count: 1 API server + 1 engine core + 4 workers = 6 processes.
A server with tensor parallel size 4 runs six processes: one API server, one engine core, and four GPU workers. Read from the top down. Each worker owns one GPU.

Why not threads

Threads in one Python process share the interpreter lock, so they would not remove the wait. Separate processes also let vLLM restart or replace one part. The repository also has an experimental front end written in Rust. It replaces the Python API server and talks to the same engine core over the same sockets.

Separate processes have a cost: each request and each output is copied across a process boundary. The copy is small next to a model step, so the design accepts it. An environment variable, VLLM_ENABLE_V1_MULTIPROCESSING=0, runs the engine core in the API server process instead. The reproducibility guide uses it for offline runs, because one process makes the order of scheduling repeatable.

vLLM 0.31.0 · db9527a · 2026-10-05 · Edition 2026.10