Choose tensor, pipeline, or data parallelism

One GPU is not enough. When you finish this section, you can pick the parallel layout from a short decision tree.

Answer three questions

When the model needs more than one machine:

• Set --pipeline-parallel-size to the number of machines, and --tensor-parallel-size to the GPUs in each machine.
• The tensor-parallel size must divide the attention-head count of the model.
• If the GPUs in each machine do not divide it, use --tensor-parallel-size 1 and set the pipeline size to the total GPU count.

Know what each layout costs

Tensor parallelism also splits the weights and the KV cache, so each GPU has more room for the cache. When the weights fill most of one GPU, two GPUs can hold several times the KV pool of one.

Check the link between GPUs

Tensor parallelism sends data after every layer. On GPUs joined only by PCIe, such as two L4s in one host, that traffic can cost more than the split saves. The vLLM documentation suggests pipeline parallelism on such a machine, because it sends data only at stage boundaries. Pipeline parallelism also accepts a GPU count that does not divide the model evenly. Measure the layouts against each other on such a machine before you pick.

Not measured on our hardware. We have not run vLLM on two L4s with tensor parallelism or data parallelism.

Start a second machine

vLLM runs one server across machines with Ray, the default for more than one machine, or with its own multiprocess launcher. We did not run either. The steps below come from the vLLM 0.31.0 documentation, for two machines with 8 GPUs each.

First, check that the GPUs can talk to each other. The vLLM troubleshooting guide has a short test script that runs NCCL and Gloo all-reduce calls. Save it as test.py on each machine. Run it on both machines at once, with NODE_RANK=0 on the first machine and NODE_RANK=1 on the second:

NCCL_DEBUG=TRACE torchrun --nnodes 2 --nproc-per-node 8 \
  --rdzv_backend=static --rdzv_endpoint=<head-ip>:29400 \
  --node-rank $NODE_RANK test.py

When the network and the drivers work, each process prints four success lines, the first of them PyTorch NCCL is successful!. The guide text names a different line, so trust the lines that the script prints.

With Ray. The vLLM repository has a run_cluster.sh script that starts the vLLM image and a Ray node on each machine. Give each machine its own VLLM_HOST_IP, on a private network. On the first machine, run:

bash run_cluster.sh vllm/vllm-openai:v0.31.0 <head-ip> --head ~/.cache/huggingface -e VLLM_HOST_IP=<head-ip>

On the second machine, run:

bash run_cluster.sh vllm/vllm-openai:v0.31.0 <head-ip> --worker ~/.cache/huggingface -e VLLM_HOST_IP=<worker-ip>

Keep both shells open, because closing one stops the cluster. Then, inside a container, check that Ray sees every node and GPU:

ray status

Start the server on one machine only. It uses the GPUs of every machine:

vllm serve Qwen/Qwen3-30B-A3B-Instruct-2507 \
  --tensor-parallel-size 8 --pipeline-parallel-size 2 \
  --distributed-executor-backend ray

Without Ray. Run vllm serve on both machines with the same model and sizes. Add --nnodes 2 --node-rank 0 --master-addr <head-ip> on the first machine. Add --nnodes 2 --node-rank 1 --master-addr <head-ip> --headless on the second machine, which then runs workers and no API server.

The traffic between the machines is not encrypted, and it can run code on the other side. Keep it on a private network that untrusted machines cannot reach.

Count the processes

With data parallelism, vLLM starts one engine core for each copy, and one coordinator that keeps them in step. With tensor parallelism, it starts one worker for each GPU under one engine core. The process page gives the count.

Use context parallelism for very long prompts

For prompts of hundreds of thousands of tokens, vLLM can split one sequence across GPUs with prefill and decode context parallelism. Use it only when one long request is the problem, not when many short ones are.