SGLang as an RL rollout engine

You train a model with GRPO. Every step, an inference server writes thousands of answers, a reward function scores them, and the trainer updates the weights. Then the server must answer the next prompts with the new weights, on the same GPUs that the trainer just used. A serving engine built for chat never had to give its memory back or change its weights while it runs. It never had to match another program's numbers to the last bit.

When you finish this section, you can describe one RL step with SGLang as the rollout engine and name the SGLang endpoints each phase calls. You can also choose a weight transfer method and say what "true on-policy" covers and what it does not. The example is Miles v0.1.2 (commit a22fd824), the RL framework that RadixArk maintains next to SGLang.

The receipt for training on stale numbers

Miles is a fork of slime, an RL framework from THUDM, and keeps slime's history. RadixArk describes it as "Enterprise-Grade Reinforcement Learning for Large-Scale Model Post-Training". The docs never define "enterprise". In the code it means three things: correctness checks between rollout and training, recovery from engine faults, and observability.

Each of those exists because RL changes what an inference server is. The server's output is the training data. The log probabilities it reports are the "old policy" in the PPO and GRPO ratio. So any numerical difference between SGLang kernels and the trainer's kernels becomes a bias in the gradient. Miles is the receipt for training on numbers that the server did not produce.

The loop that borrows the server

The training driver is short on purpose. At v0.1.2, train.py has 175 lines. These are the lines of one step when the trainer and SGLang share GPUs:

for rollout_id in range(args.start_rollout_id, args.num_rollout):
    await inference_controller.prepare_rollout(rollout_id)
    rollout_data_pack = await rollout_executor.get(rollout_id)

    if args.offload_rollout:
        ...
            offload_tags = [GPU_MEMORY_TYPE_CUDA_GRAPH]
            if "kv_cache" in args.offload_rollout_level:
                offload_tags.append(GPU_MEMORY_TYPE_KV_CACHE)
            if "weight" in args.offload_rollout_level:
                offload_tags.append(GPU_MEMORY_TYPE_WEIGHTS)
            await inference_controller.offload(tags=offload_tags)
    ...
        await actor_model.train(rollout_id, rollout_data_pack)
    ...
            await offload_train()
            if args.offload_rollout:
                await inference_controller.onload_weights()
        await update_weights(args, actor_model, rollout_executor, inference_controller, rollout_id=rollout_id)
        if args.offload_rollout:
            await inference_controller.onload_kv()

Read it as a rental. SGLang generates and scores the rollout. SGLang releases its CUDA graphs, KV cache, and weights. The trainer loads, trains, and offloads itself. SGLang gets its weights back, receives the new weights, and then gets its KV cache and CUDA graphs back. Figure 7.2b shows which memory is on the GPU in each phase.

What stock SGLang gives the trainer

The rental works through SGLang HTTP endpoints that exist in stock v0.5.22. The RL guide of SGLang states the design rule. SGLang acts as a library that the trainer calls, not as a framework that owns the loop. Miles drives each engine as an ordinary sglang serve process over HTTP.

Memory release: POST /release_memory_occupation frees GPU memory by tag, and POST /resume_memory_occupation gives it back. The server needs --enable-memory-saver, which is off by default. The tags are kv_cache, weights, and cuda_graph. The docs list only the first two, and the code accepts all three, so the code wins. The memory saver frees physical memory but keeps the virtual addresses, so captured CUDA graphs stay valid without a new capture. The call asserts that the server is idle, so pause generation first.

Pause and resume: POST /pause_generation takes a mode. SGLang's default is abort, which ends every running request. Miles uses retract by default, which puts running requests back in the waiting queue so they prefill again after the update. A third mode, in_place, keeps their KV cache from the old weights.

The weight-update session: begin_weight_update, then one or more update_weights_from_* calls, then end_weight_update. The session exists because quantized layers need a one-time repack after loading. Doing it once per refit is cheaper than once per bucket. After the session, update_weight_version stamps the new version. A request that lived through a refit reports weight_versions spans in its meta_info, so you can see which tokens came from which weights. Figure 7.2 shows one update with the default transfer.

Four ways to move the weights

Miles picks the transfer with --update-weight-transfer-mode:

With --colocate, Miles ignores this flag and always sends CUDA IPC handles through /update_weights_from_tensor. A team that sets p2p on a colocated run measures the wrong thing and never knows.

The Miles team reports P2P timings on H100 clusters with a 1 GB bucket. Qwen3-235B-A22B on 8 nodes went from 10,754 ms with NCCL broadcast to 3,162 ms, 71% less. Kimi-K2 on 32 nodes went from 53,279 ms to 7,227 ms, 86% less. On one node P2P was slower: GLM-4.7-9B-Flash took 68.6% longer and a 4-layer GLM-5 test took 72.2% longer. The reason is in the design. Broadcast sends the full model to every engine rank, and with expert parallelism each rank needs only a fraction of it.

RadixArk reports that about 2% of BF16 parameters change in one step. For GLM-4.7-Flash the disk-delta payload fell from 62.4 GB to 0.69 to 0.83 GB per update, with a generation pause of 3 to 5 s. These are their numbers from the Miles v0.1 post of 2026-08-18. We did not reproduce them.

Our position: start colocated on one node with the CUDA IPC path. Move to separate GPUs only when rollout time dominates the step. Choose p2p only for multi-node MoE, where the vendor table shows the gain, and keep broadcast everywhere else.

True on-policy, and where it stops

Here the serving engine and the trainer must agree on a number. Give SGLang and Megatron or FSDP the same weights and the same tokens. Their kernels can still return different log probabilities, because their reductions run in a different order. Then the PPO ratio mixes "the policy changed" with "the kernels differ". Miles reports the gap as train/train_rollout_logprob_abs_diff, which should be exactly 0 in true on-policy mode.

Two SGLang facts matter first. SGLang returns log probabilities after temperature scaling unless the request is greedy or SGLANG_RETURN_ORIGINAL_LOGPROB is set, as section 4.6 explains. Sometimes the rollout samples at a temperature other than 1, and the trainer computes log probabilities at temperature 1. Then the trainer sees a mismatch that no kernel causes.

Second, --rl-on-policy-target accepts one value at v0.5.22, fsdp, and it forces deterministic inference. That mode replaces matrix multiply, softmax, and norm kernels with batch-invariant versions and forces the PyTorch sampler. It also turns off the radix cache unless the attention backend is fa3, fa4, triton, or ascend.

Miles builds on that with a launch plan that sets matching flags on both sides. The coverage is narrow. At v0.1.2 the only model profile is Qwen3 dense: Qwen3-0.6B, Qwen3-4B, Qwen3-4B-Base, and Qwen3-4B-Instruct-2507. Three open issues in the Miles repository show where bitwise parity still breaks:

• Issue #1485: Qwen3-4B down_proj outputs differ at tensor parallel 2. SGLang sums 38 partial results in two groups of 19, and Miles sums them in a pairwise tree. BF16 addition is not associative.
• Issue #3714: deterministic serving pins the NCCL channel limit to 8 while the trainer uses 24. Weight transfer then fails with Message truncated : received 6144 bytes instead of 2048.
• Issue #3277: after a retract, re-prefill under the new weights overwrites the recorded MoE routes. The response then carries new routes with old log probabilities.

Our position: treat true on-policy as a diagnostic for Qwen3 dense models, not a switch for every run. Log the log-probability gap on every run anyway. A large gap means a conversion or kernel problem, and no RL algorithm fixes that.

What we measured with Miles on a GB10

We ran GRPO with the radixark/miles:latest image on one NVIDIA GB10. The image held Miles commit 08752a12 and SGLang 0.5.22.dev46+g7b6993a. The model was Qwen/Qwen3-0.6B on GSM8K, with FSDP and SGLang colocated on the one GPU. Each of the 12 steps sampled 32 prompts 8 times, up to 1,024 response tokens, at temperature 1. SGLang had a memory fraction of 0.4.

metrics.py:94 - perf 0: {'rollout/raw_reward_unfiltered': 0.62890625, 'rollout/num_training_samples': 256, 'rollout/episode_raw_reward': 0.62890625, 'rollout/episode_response_length/mean': 739.92578125, 'rollout/episode_response_length/median': 759.5, 'rollout
{'train/loss': -4.656612873077393e-10, 'train/pg_loss': -4.656612873077393e-10, 'train/entropy_loss': 0.0, 'train/pg_clipfrac': 0.0, 'train/ppo_kl': 0.0, 'train/ess_ratio': 1.0, 'train/train_rollout_logprob_abs_diff': 0.015336457639932632, 'train/train_rollout_kl': 0.0006840902497060597, 'train/grad_norm': 0.5216571092605591, 'train/lr-pg_0': 1e-06, 'train/step': 0}
timer.py:42 - Timer update_weights end (elapsed: 0.7s)
[2026-10-17 15:55:01] post-warmup freeze_gc failed

We measured the step time by phase. The SGLang rollout took a median of 57.2 s, at about 3,113 tokens per second. The trainer then recomputed log probabilities in 25.2 s and trained in 80.3 s. The weight update into SGLang took 0.65 s through CUDA IPC. Releasing the SGLang memory took a median of 0.8 s and at most 4.0 s, and taking it back took 0.9 s. Raw runs

The rollout was not the bottleneck at this size. For a 0.6B model on one GPU, a faster serving engine would save at most about a third of the step, 35 percent. The weight update and the memory release, which much of this section is about, cost 2.4 s. The step took about 165 s, summed from the phase medians.

The log-probability gap stayed between 0.0143 and 0.0163 in all 12 steps. That is the number true on-policy mode exists to drive to 0, and plain mode does not move it. The raw reward went up and down between 0.586 and 0.758. Twelve steps are too few to show learning, so read the run as a timing and a gap, not as a training curve.

Forty-eight steps

We then ran the same job for 48 steps, with the same settings: memory fraction 0.4 and temperature 1, first with the default seed. Over those steps the policy started to learn, and the rollout got faster as it did.

metrics.py:94 - perf 47: {'rollout/raw_reward_unfiltered': 0.79296875, 'rollout/num_training_samples': 256, 'rollout/episode_raw_reward': 0.79296875, 'rollout/episode_response_length/mean': 421.73046875, 'rollout/episode_response_length/median': 369.5, 'rollou
{'train/loss': -3.4924596548080444e-09, 'train/pg_loss': -3.4924596548080444e-09, 'train/entropy_loss': 0.0, 'train/pg_clipfrac': 0.0, 'train/ppo_kl': 0.0, 'train/ess_ratio': 1.0, 'train/train_rollout_logprob_abs_diff': 0.014126655645668507, 'train/train_rollo
Subprocess scheduler_0 (pid=3263) crashed with exit code -15. Triggering SIGQUIT for cleanup...
Job 'raysubmit_MKE9XLVeZ87U7iQV' succeeded

We measured a mean raw reward of 0.636 over the first 8 steps and 0.758 over the last 8. The responses got shorter. Their mean length fell from 724 to 447 tokens, and the share of samples cut at the 1,024-token limit fell from 0.33 to 0.08. SGLang had fewer tokens to make, so the rollout fell from 60.2 s to 32.7 s per step. Its rate went up from 3,084 to 3,501 tokens per second. Raw runs

From one step to the next the reward changed by as much as 0.207, from 0.801 at step 30 to 0.594 at step 31. Read the trend from the 8-step means, not from one step.

The log-probability gap did not move. It was 0.0153 over the first 8 steps and 0.0146 over the last 8, and it stayed between 0.0138 and 0.0162 in all 48. In 48 steps of this 0.6B run, training did not close the gap. Miles attributes the gap to differences between the SGLang and FSDP kernels. True on-policy mode is the part of Miles that works on it.

The last log lines also need a note. The SGLang scheduler printed crashed with exit code -15, and the Ray job then reported success. Exit code -15 means the scheduler got SIGTERM, here at the end of a finished job. Alert on the job status, not on that line alone.

Two log lines are worth knowing before you trust a run. About 30 percent of the samples in the 12-step run hit the 1,024-token limit, and 20 percent over the 48 steps. So the reward includes answers cut off before the end. And SGLang printed post-warmup freeze_gc failed once at startup. Training ran normally after it, so the warning cost nothing here.

Two more seeds

We ran the same 48 steps twice more, with MILES_SEED=2 and MILES_SEED=3. The variable sets the train seed and the rollout seed. The GB10 shut down three times under this load on 2026-10-18, and its boot list shows three short boots that day. So these runs had a guard. A script paused the container when the GPU reached 84 C, or 80 C in the first 30 minutes. It resumed the container after about 45 to 60 seconds, at 72 C or below.

Over the 9.3 hours of seeds 2 and 3, the container was paused for about 58 percent of the time. A pause stops the process and does not change its arithmetic. So we read the reward, the lengths, the cut share, and the log-probability gap as valid. We did not test this with a paused and an unpaused run of the same seed. The rollout times are not, and we do not report them for seeds 2 and 3.

metrics.py:94 - perf 47: {'rollout/raw_reward_unfiltered': 0.6484375, 'rollout/num_training_samples': 256, 'rollout/episode_raw_reward': 0.6484375, 'rollout/episode_response_length/mean': 514.0390625, 'rollout/episode_response_length/median
{'train/loss': -5.122274160385132e-09, 'train/pg_loss': -5.122274160385132e-09, 'train/entropy_loss': 0.0, 'train/pg_clipfrac': 0.0, 'train/ppo_kl': 0.0, 'train/ess_ratio': 1.0, 'train/train_rollout_logprob_abs_diff': 0.014963770285248756,
Job 'raysubmit_y4umCbEa8Qp5Bbad' succeeded

We measured the same result with all three seeds. The reward went up, the responses got shorter, and fewer of them were cut. The log-probability gap stayed between 0.0136 and 0.0164 in all 144 steps. The largest change in reward from one step to the next was 0.207, 0.160, and 0.203 for the three seeds. Raw runs

True on-policy did not start on a GB10

We then set --true-on-policy-mode with the flags of the Miles Triton example. SGLang started, and the trainer stopped 61 seconds after the launcher began.

[2026-10-17 16:28:41.340 launcher] common.py:290 - EXEC: mkdir -p /root/lab/mode
ValueError: Specified `attn_implementation="triton"` is not supported. The only possible arguments are `attn_implementation="eager"`, `"paged|eager"`, `"attn_implementation=flash_attention_2"`, `"attn_implementation=paged|flash_attention_2"`, `"attn_implementa
[2026-10-17 16:29:42.606 trainer_controller_actor] cell.py:251 - ft op=execute phase=fail cell=trainer-engine-actor-00000 fn=init elapsed_s=0.7

The flag value came from the wrong example. In the FSDP actor, the SGLang Triton attention bridge turns on only when torch.version.hip is set, which means AMD GPUs. On an NVIDIA GPU the value goes straight to transformers, which has no attention named triton.

The only true on-policy contract in Miles pairs the SGLang fa3 backend with flash_attention_3 in FSDP. We ran the job again with the contract flags to see where it stops.

[2026-10-17 21:03:03] Enable deterministic inference because of legacy rl_on_policy_target.
AssertionError: FlashAttention v3 Backend requires SM>=80 and SM<=90. Please use `--attention-backend flashinfer`.
[2026-10-17 21:03:24] Received sigquit from a child process. It usually means the child failed.
[2026-10-17 21:03:30.637 rollout_executor] http_utils.py:241 - Error: Server error '503 Service Unavailable' for url 'http://172.17.0.3:20035/generate'
[2026-10-17 22:06:46.137 rollout_executor] http_utils.py:241 - Error: Server error '503 Service Unavailable' for url 'http://172.17.0.3:20035/generate'

SGLang stopped 21 seconds after it started. Its FA3 backend accepts GPUs from SM 8.0 to SM 9.0, and the GB10 is SM 12.1. With the Miles image we ran, at commit 08752a12, the only true on-policy contract needs FA3, so it cannot start on a GB10.

What happened next is the part to remember. The rollout engine was gone, but the Miles job did not stop. Its rollout loop kept sending requests to the router, which answered 503 Service Unavailable. The log holds about 200,000 of those errors over 63 minutes, until we stopped the container. The job never reported a failure. We did not record the GPU during that hour.

Our position on the GB10 result: before you plan a true on-policy run, check that your GPU and image have both kernels of the contract. Then watch the SGLang server log, not only the trainer, because a dead rollout engine can look like a slow one. In the image we ran, at Miles commit 08752a12, that contract means FlashAttention 3 on both sides, a path written for Hopper GPUs.

Miles runs on its own SGLang

Miles does not use the SGLang release that the rest of this manual describes. Its Docker image builds on lmsysorg/sglang:v0.5.21 and checks out the sglang-miles branch. The v0.1.2 release lock pins SGLang commit b84bec5761a6. The Miles docs say it plainly: stock sglang or upstream Megatron-LM installed next to Miles does not work.

At least four things that Miles calls do not exist at v0.5.22: the /pull_weights endpoint for disk-delta, register_lora_adapter, get_parallelism_info, and the --true-on-policy-contract flag.

That branch is a fair summary of this section. RL asks a serving engine for things that serving users never asked for. It must give memory back and change weights in a session. It must report which weights produced each token and match another program bit for bit. Some of those requests are now in stock SGLang. The rest live on a branch until they are ready.

A real incident: issue #860 in the Miles repository reports that /release_memory_occupation "silently does nothing" for models with sliding-window attention. The sliding-window KV pool turns the memory saver off in its constructor, so a colocated run runs out of memory when Megatron starts. One user saw 30.19 GB of KV pool stay resident on DeepSeek-V4-Flash. The issue was open on 2026-10-16. Check that every memory tag really frees memory on a new model before you trust colocation.