Use vLLM as an RL rollout engine

Your RL trainer must generate rollouts and then update the weights. When you finish this section, you can name the control routes an RL framework calls.

The loop

An RL step has two halves. The rollout half generates answers with the current weights. The training half computes a loss and updates the weights. Then the generator needs the new weights before the next rollout.

On one set of GPUs, the two halves take turns on the same memory. The generator must give its memory back during training and take it again after. The vLLM server provides the pieces for this, and many RL frameworks call them. The vLLM documentation lists verl, OpenRLHF, TRL, NeMo-RL, SkyRL, Prime-RL, and others among its users.

The pieces

The routes below exist only when you start the server with VLLM_SERVER_DEV_MODE=1. They free memory, replace weights, and run methods on every worker. Never expose a server in this mode outside the network of the trainer.

Sleep mode. Start the server with --enable-sleep-mode. A call to /sleep frees GPU memory, and /wake_up takes it back. At the deeper of its two levels, sleep drops the weights too, which suits a step where the trainer sends new weights anyway. --sleep-preserve-parameter-names keeps frozen weights on the GPU, so they do not move each step.

Pause and resume. /pause stops new generation, and /resume starts it again. A pause can abort the running requests, wait for them, or keep them.

Weight updates. The trainer starts a transfer with /init_weight_transfer_engine, then calls /start_weight_update, sends the weights, and calls /finish_weight_update. The transfer can run over NCCL between GPUs or through CUDA memory handles on one machine. /update_weight_version records which version the next rollouts use.

Collective calls. /collective_rpc runs a method on every worker. Frameworks use it for setup that vLLM does not cover.

External launch. With --distributed-executor-backend external_launcher, a launcher such as torchrun starts the vLLM workers inside the trainer's processes. Some colocated frameworks use this path.

Try sleep and wake by hand

Start a server in development mode with sleep mode on. --host 127.0.0.1 keeps the control routes on your own machine:

VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen3-8B --enable-sleep-mode --host 127.0.0.1 2>&1 | tee server.log

In a second terminal, put the server to sleep, check it, and wake it:

curl -X POST 'http://localhost:8000/sleep?level=1'
curl http://localhost:8000/is_sleeping
curl -X POST 'http://localhost:8000/wake_up'
curl http://localhost:8000/is_sleeping

The two POST routes return HTTP 200 with an empty body. The first /is_sleeping call returns {"is_sleeping":true}, and the second returns {"is_sleeping":false}. Run nvidia-smi between the calls to see the GPU memory fall during the sleep and rise after the wake.

We ran these calls at level 1 with the small Qwen3-0.6B on an L4. Each POST returned 200 with an empty body, and the sleep took 1.9 s. The GPU memory fell from 14,861 MiB to 4,245 MiB and came back after the wake. The log line Sleep mode freed 10.37 GiB memory confirms the sleep.

Know the sleep mode problems at 0.31.0

Three problems with sleep mode are known at 0.31.0. We did not reproduce them ourselves:

• A level 2 wake does not reload a speculative draft. Issue 52479 reports that the weight reload covers only the target model. The target model then rejects every draft token, so decode falls to about the speed with no speculation. The output stays correct, and no error appears. The issue is open.
• A level 2 sleep clears calibrated KV scales. On compressed-tensors checkpoints with an FP8 KV cache, the first request after a reload of the weights returned NaN logprobs and a wrong first token. Release 0.32.0 fixes this.
• A weight update during sleep stops the engine. Issue 57853 reports a segfault when the trainer sends weights after /sleep and before /wake_up. The API server stays up, and every later request fails. The issue is open.

So after each level 2 wake, reload the weights, reset the prefix cache, and read the speculative acceptance counters. Always wake the server before the trainer sends weights.

Check the rollout logprobs

The trainer often compares the logprobs that vLLM returned with the logprobs it computes itself. The two never match exactly, because the kernels differ. A small, steady gap is normal. A gap that grows means the weights did not reach the generator. Track the gap every step. The reproducibility page explains why the same weights still give different numbers.

Before you read the gap, check which logprob vLLM returns. With top-p or top-k, the two values differ for most tokens.

Logprobs mode: the server option --logprobs-mode, which picks the value that vLLM returns for each token. The default is raw_logprobs, the logprob over the full vocabulary. processed_logprobs gives the logprob after the sampling settings change the distribution. Two more modes, raw_logits and processed_logits, return logits.

We checked both logprob modes on the second GPU of our L4 node on 2026-10-23. We used Qwen/Qwen3-0.6B in BF16, 16 prompts of 48 tokens, temperature 1, and top-p 0.9. A float32 Transformers pass over the same tokens gave the reference. That GPU is shared and has no clock lock, but the clock does not change these values.

engine and mode                                  returned minus reference, median   median absolute
vLLM --logprobs-mode raw_logprobs                0.0000                             0.005
vLLM --logprobs-mode processed_logprobs          +0.0527

• raw_logprobs matched the full-vocabulary logprob, with a median absolute gap of 0.005 from the kernels.
• processed_logprobs was higher by minus the log of the probability mass that top-p kept. The residual against that formula had a median of 8e-9.
• At top-p 0.9, the kept mass was below 0.99 for most sampled tokens, and the median shift was about 0.053.

So compare raw_logprobs with a trainer that scores over the full vocabulary. Compare processed_logprobs only with a trainer that scores over the same kept set.

Not measured on our hardware. We ran only that one level 1 check and the logprob check, both with small models. We have not timed a level 2 sleep or a weight update. Our Miles 101 manual measures the same loop with SGLang as the generator.