We start from an empty folder and end with one finished GRPO step, logged to a file. The model is Qwen3-0.6B and the data is GSM8K, so the run fits on one GPU and each step takes a few minutes.

When you finish this section, you can start a GRPO run on one GPU. You can also find the log lines that prove the first step finished.

Our result: in a 12-step run of Qwen3-0.6B on one GB10, the first step took 233 s and the median step 167 s.

Check what you need

You need one NVIDIA GPU, Docker, and the NVIDIA container toolkit. Run this check first:

docker run --rm --gpus all radixark/miles:latest nvidia-smi --query-gpu=name,memory.total --format=csv

You should see your GPU name. If Docker says it cannot select a device driver, install the NVIDIA container toolkit and restart Docker.

name, memory.total [MiB]
NVIDIA GB10, [N/A]

Our GPU was a DGX Spark GB10 with 128 GB of shared memory. We did not test a smaller card. Section 2.1 covers model size and memory.

Pull the image and record what you pulled

The latest tag moves. Record the image digest and the Miles commit before the run, so you can compare runs later.

docker pull radixark/miles:latest
docker image inspect --format "{{index .RepoDigests 0}} {{.Architecture}}" radixark/miles:latest
docker run --rm --entrypoint git radixark/miles:latest -C /root/miles log -1 --format="%h %cs %s"
docker run --rm --entrypoint /opt/sglang/bin/python3 radixark/miles:latest -c "import sglang;print(sglang.__version__)"

radixark/miles@sha256:c64fa344e3a86715f03c0a1b6a422c474e0035355591a2f6da87bf52a74c52da arm64
08752a1283 2026-10-15 fix(ci): keep host networks out of the builder disk cleanup (#4130)
0.5.22.dev46+g7b6993a

The image is built for arm64 and amd64. It holds its own build of SGLang, which is the rollout engine. Use that build and do not install another SGLang into the image.

Make a lab folder

The container reads the launcher script, the model, and the data from one folder that you mount at /root/lab. Create it and download the launcher script of this manual into it:

mkdir -p ~/miles-lab/runs ~/.cache/huggingface
curl -fsSL -o ~/miles-lab/miles_grpo.py https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/manuals/miles-101/capture/gpu/miles_grpo.py

If you have a clone of this repository, run cp manuals/miles-101/capture/gpu/miles_grpo.py ~/miles-lab/ from its root instead. The second folder is the model cache that the run mounts. If it does not exist, Docker creates it as root, and later downloads on the host fail.

The launcher downloads Qwen/Qwen3-0.6B and the zhuzilin/gsm8k dataset into the folder on its first run. Then it builds the Miles command and starts it.

Read the settings the launcher uses

The launcher passes 36 flags to Miles. Most of them keep their defaults. These are the ones that shape the run:

Group: the answers that one prompt gets in one step. GRPO scores each answer against the mean reward of its group.

The rollout batch size counts prompts, and the global batch size counts samples. With 32 prompts and 8 answers each, one rollout gives 256 samples, so each step makes one optimizer update.

Add flags to a run

The launcher reads its settings from environment variables. Set each one with -e on the docker run line:

Every recipe in this manual adds its flags through MILES_EXTRA_ARGS. For example, this line adds the top-p and top-k settings of section 4.4:

-e MILES_EXTRA_ARGS="--rollout-top-p 0.95 --rollout-top-k 64"

When a flag is in the launcher and also in MILES_EXTRA_ARGS, the value in MILES_EXTRA_ARGS wins, because Miles keeps the last value. Our runs with an engine memory fraction of 0.25 set it that way. There is one exception. With MILES_TRUE_ON_POLICY=1, the launcher adds its true on-policy flags after MILES_EXTRA_ARGS, so those flags win. Set the attention backends of that mode with MILES_SGLANG_ATTENTION and MILES_FSDP_ATTENTION instead.

Start the run

Run the container with the GPU, the lab folder, and your model cache mounted:

cd ~/miles-lab
docker run --rm --name miles-plain --gpus all --ipc=host --shm-size 32g --ulimit memlock=-1 \
  -v "$HOME/miles-lab:/root/lab" -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  -e MILES_NUM_ROLLOUT=12 -e MILES_RUN=plain \
  radixark/miles:latest python /root/lab/miles_grpo.py > runs/plain.log 2>&1

Our 12-step run took 35 minutes on the GB10, from the first log line to the last. Start the watchdog from section 1.2 in a second terminal before you leave the run alone.

To end a run early, stop its container from another terminal:

docker stop miles-plain

Give each run its own container name, MILES_RUN, and log file, so that two runs never share a log.

Read the startup lines

Open the log in another terminal with tail -f runs/plain.log. The run starts Ray, then the rollout engine, then the trainer. These lines tell you each part came up:

[2026-10-17 15:53:16.726 launcher] common.py:290 - EXEC: export PYTHONUNBUFFERED=1 && ray start --head --node-ip-address 127.0.0.1 --num-gpus 1 …
(CommandActor pid=2546) [2026-10-17 15:54:38] Load weight end. elapsed=5.85 s, type=Qwen3ForCausalLM, avail mem=94.82 GB, mem usage=1.40 GB.
(CommandActor pid=2546) [2026-10-17 15:54:40] KV Cache is allocated. dtype: torch.bfloat16, #tokens: 345645, K size: 18.46 GB, V size: 18.46 GB

The engine loaded 1.40 GB of weights and kept room for 345,645 tokens of KV cache. One step needs at most 256 answers of 1,024 tokens each, plus their prompts.

If no KV Cache is allocated line appears within five minutes, the rollout engine did not start. Read the lines above it for an Error or an AssertionError, and see section 7.1.

Find the first finished step

A step ends when the trainer logs a step 0 line and a perf 0 line. Search for them:

grep -c " - step [0-9]*: {" runs/plain.log

The count is the number of finished steps. Our first step logged these lines:

(FSDPTrainRayActor pid=2551) [2026-10-17 15:56:48.271 actor_cell00000_rank00000] log_utils.py:125 - rollout 0: {'rollout/response_lengths': 739.92578125, … 'rollout/raw_reward': 0.62890625, …
(FSDPTrainRayActor pid=2551) [2026-10-17 15:58:15.687 actor_cell00000_rank00000] log_utils.py:554 - step 0: { … 'train/train_rollout_logprob_abs_diff': 0.015336457639932632, …
(FSDPTrainRayActor pid=2551) [2026-10-17 15:58:15.688 actor_cell00000_rank00000] perf.py:56 - perf 0: {'perf/sleep_time': 2.1116080284118652, … 'perf/update_weights_time': 0.6647534370422363, … 'perf/step_time': 233.29494333267212, …

In this 12-step run, the first step took 233 s and the median step took 167 s. In a typical step, the rollout took 57 s and the training phase took 105 s, and 25 s of that computed log probabilities. The weight update from the trainer to the engine took 0.65 s.

The two figures on this page use our 48-step run of the same setup. Its median step was 152.6 s and its median rollout 47.5 s. Its later steps wrote shorter answers, so its medians are lower.

Section 1.3 explains how to read these numbers over the first ten steps.

We left a run alone for an hour. When we came back, the GPU was idle and the log had grown to 147 MB. The rollout engine had died during startup, and the run had sent requests to it ever since.

When you finish this section, you can recognize a run whose engine died. You can also start a watchdog that stops that run within a few minutes.

Our result: a run whose engine died logged 200,533 HTTP 503 errors in 63 minutes and never failed. The watchdog stopped a test of the same error in 183 s. A replay of a healthy run showed that six checks in a row are too few, so the script waits for ten.

What a dead engine looks like

The run below used a setting that this GPU cannot run. The rollout engine stopped during startup. The trainer did not stop, and it sent requests to the engine for 63 minutes:

[2026-10-17 21:03:03] Enable deterministic inference because of legacy rl_on_policy_target.
AssertionError: FlashAttention v3 Backend requires SM>=80 and SM<=90. Please use `--attention-backend flashinfer`.
[2026-10-17 21:03:24] Received sigquit from a child process. It usually means the child failed.
[2026-10-17 21:03:30.637 rollout_executor] http_utils.py:241 - Error: Server error '503 Service Unavailable' for url 'http://172.17.0.3:20035/generate'
[2026-10-17 22:06:46.137 rollout_executor] http_utils.py:241 - Error: Server error '503 Service Unavailable' for url 'http://172.17.0.3:20035/generate'

$ grep -c "503 Service Unavailable" true-on-policy-fa3.log
200533

The run wrote 200,533 error lines and never reported a failure.

Router: the small service that forwards each rollout request to an engine. Section 8.6 explains it.

503 Service Unavailable: the HTTP status that the router returns when no rollout engine can take the request.

Why the watchdog waits for ten checks

In our 17 runs that ended normally, the log held no 503 Service Unavailable line at all. Our 16-answer run of section 3.1 logged 22,357 of them. They came in bursts of up to about 60 s, with up to 2,209 in one minute. The client retried each request, and the run finished all its steps.

In the failed run, the errors never stopped. Every full minute held at least 2,669 of them, for 63 minutes.

A count alone cannot tell these two runs apart, but the time can. Every 30 s, the watchdog counts the errors of the last 60 s. It stops the run only when that count is above 100 in a set number of checks in a row.

$ python3 watchdog_replay.py out-gb10/e2-n16-503-per-second.txt
healthy run: 22357 errors from 2026-10-22 07:16:29, checks every 30 s over the last 60 s, limit 100
checks  healthy offsets stopped  most checks in a row  dead engine stopped after
     6  17 of 30                    6                     153 to 182 s
     8   0 of 30                    6                     213 to 242 s
    10   0 of 30                    6                     273 to 302 s

We measured the rule on the real error times of that healthy run. With 6 checks, it stopped the run in 17 of the 30 cases, because two bursts came 45 s apart. No case reached more than 6 checks in a row. With 10 checks, no case stopped the healthy run, and a dead engine is stopped in about five minutes. Raw runs

Start the watchdog

The script reads the last 60 seconds of the container log every 30 seconds. It stops the container when the count passes the limit in a set number of checks in a row.

If Docker itself fails, the script says so. It exits with 2 when it cannot list the containers. A failed log read does not count as a check. The script also exits with 2 for a bad limit or number of checks: a value that is not a whole number, or 0 checks.

The third value is the number of checks in a row. Its default is 10, which a healthy run did not reach in our replay:

#!/bin/sh
name=${1:-miles-plain}
limit=${2:-100}
checks=${3:-10}
for value in "$limit" "$checks"; do
  case $value in
    ''|*[!0-9]*|??????????*) echo "limit and checks must be whole numbers below one billion"; exit 2;;
  esac
done
if [ "$checks" -lt 1 ]; then
  echo "checks must be 1 or more"
  exit 2
fi
strikes=0
while :; do
  if ! running=$(docker ps -q --filter "name=^${name}$"); then
    echo "$(date -u +%FT%TZ) $name: docker ps failed, the watchdog cannot see the run and stops"
    exit 2
  fi
  [ -n "$running" ] || break
  if logs=$(docker logs --since 60s "$name" 2>&1); then
    errors=$(printf '%s\n' "$logs" | grep -c "503 Service Unavailable")
    if [ "$errors" -gt "$limit" ]; then
      strikes=$((strikes + 1))
    else
      strikes=0
    fi
  else
    echo "$(date -u +%FT%TZ) $name: docker logs failed, this check does not count"
  fi
  if [ "$strikes" -ge "$checks" ]; then
    echo "$(date -u +%FT%TZ) $name: more than $limit errors from the rollout engine in each of $checks checks, last $errors in 60 s, stopping the run"
    docker stop "$name"
    exit 1
  fi
  sleep 30
done
echo "$(date -u +%FT%TZ) $name: the run ended"

Save it as ~/miles-lab/watchdog.sh, or download the same script:

curl -fsSL -o ~/miles-lab/watchdog.sh https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/manuals/miles-101/capture/gpu/watchdog.sh

Start it in a second terminal right after the run starts, with the container name from section 1.1:

sh ~/miles-lab/watchdog.sh miles-plain 100 10

Test the watchdog first

Test it on two containers that print the error on purpose. They need no GPU. The first prints ten error lines each second and never stops, like a dead engine:

$ sh capture/gpu/watchdog.sh wd-dead 100
2026-10-22T15:41:09Z wd-dead: more than 100 errors from the rollout engine in each of 6 checks, last 578 in 60 s, stopping the run
wd-dead
$ echo $?
1

The watchdog stopped the container 183 s after it started, and exited with 1. We ran this test before we raised the default to 10, so it used 6 checks. With 10 checks, the same rule needs four more checks, so it stops about two minutes later. A script that starts runs one after another can read that exit code and skip to the next run. An exit code of 2 means that the watchdog lost Docker, and the run has no guard.

The second container prints the error for 60 s and then stays quiet for two minutes. It does this three times, like the bursts of a busy engine. The watchdog let it run to its end after 541 s, and exited with 0. Its quiet gaps of two minutes are longer than the real gaps of 45 s, so test with the replay above too.

Guard the GPU heat

Our GB10 reached 84 C again and again under hours of load. The heat guard from the front matter paused our runs 1,005 times in five days. Run it on a GB10, or on any GPU that gets hot when you leave a run alone for hours. A short run of a few steps does not need it.

Download the script that we ran, and start it in a third terminal on the host, not in the container. The rm part deletes a stop file that an earlier session can leave behind, because the script exits at once when that file exists. Delete it only when no queue of yours uses the file to stop the guard. The commands stop at the first failure, so a failed download does not start an old copy:

curl -fsSL -o ~/miles-lab/heat-guard.sh https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/manuals/miles-101/capture/gpu/heat-guard.sh \
  && rm -f ~/miles-lab/queue2.done \
  && sh ~/miles-lab/heat-guard.sh

Every 5 s, the script reads the GPU temperature with nvidia-smi. It pauses the first running container of the radixark/miles:latest image at 84 C. It resumes the container at 74 C or below after at least 45 s, or after 120 s at most. Each pause and resume goes to ~/miles-lab/runs/thermostat.log.

Stop the script with Ctrl+C. It resumes the container that it paused before it exits, so no run stays paused. In our check, Ctrl+C during a pause resumed the run within 5 s. It also stops when the file ~/miles-lab/queue2.done exists, which is how our run queue stopped it.

Read the pause log when you compare times. A step that overlapped a pause took longer for a reason that is not the work. Section 5.3 shows how to time only clean steps.

The watchdog only stops the run. To find out why the engine died, read the engine lines above the first error. Section 7.1 lists the causes we met.

The reward goes up and down in the first steps, and it is easy to read too much into one step. Five numbers in the log tell you more. Each one has a healthy range on this setup, and each one breaks in a different way.

When you finish this section, you can pull five numbers out of a run log. You can then compare them with our ranges and decide if the run is healthy.

Our result: in the first ten steps of three seeds, the reward per step was 0.53 to 0.74. The log probability gap was 0.0141 to 0.0164.

Pull the numbers out of the log

Each step writes its numbers on two lines: a rollout N line for the samples and a step N line for the training update. Go to the folder of the log from section 1.1 with cd ~/miles-lab/runs. Then use grep to read one number over the steps:

$ grep -o "'rollout/raw_reward': [0-9.]*" plain.log | head -10
'rollout/raw_reward': 0.62890625
'rollout/raw_reward': 0.59375
'rollout/raw_reward': 0.7578125
…
$ grep -o "'train/train_rollout_logprob_abs_diff': [0-9.]*" plain.log | head -10
'train/train_rollout_logprob_abs_diff': 0.015336457639932632
'train/train_rollout_logprob_abs_diff': 0.01610221341252327
'train/train_rollout_logprob_abs_diff': 0.014312651939690113
…

Change the metric name in the pattern to read any other number.

Compare with the healthy ranges

KL: the Kullback-Leibler divergence, a distance between two probability distributions. It is 0 when the two distributions are the same.

We ran the same 48-step job with three seeds. These are the values over the first ten steps of all three:

Log probability gap: the mean absolute difference between the log probability that the engine gave each sampled token and the one the trainer computes. It is never zero, because the two sides run different kernels.

From one step to the next, the reward moved by up to 0.14 in the first ten steps. Over 48 steps, it moved by up to 0.21. Read the mean over five or more steps, not one step.

Read the trend over the full run

Over 48 steps, all three seeds moved the same way. The reward rose, the answers got shorter, fewer answers hit the length limit, and the log probability gap stayed flat. Section 5.3 gives the numbers for each seed.

Know the signs of a broken run

The last row needs one more number. The rollout side writes a perf N line for each step, and it counts groups where every answer got the same reward. In our first step, rollout/zero_std/count_1 was 7 and rollout/zero_std/count_0 was 3. So 10 of the 32 groups had every answer right or every answer wrong, and GRPO learns nothing from those groups.

When you finish this section, you can estimate the memory of a run before you start it. You can also choose the largest model that fits.

Our result: on one GB10 with 128 GB of shared memory, Qwen3-1.7B fit for 4 short steps with answers cut at 256 tokens. Qwen3-4B did not fit, even with those short answers. We did not train 1.7B with the 1,024-token recipe.

Count the trainer memory first

The trainer is the large part of a colocated run. Full fine-tuning with Adam keeps the model weights, the gradients, and two optimizer values for each weight. With mixed precision that is about 16 bytes for each parameter, before activations. The engine adds its own copy of the weights and its KV cache on top.

The estimate is a lower bound. Activations, the engine, and the process overhead come on top, and the measured node memory below shows how large that overhead is.

What happened with 4B

Ray stopped both of our 4B runs in the training phase of the first step, at 115.86 and then 119.18 GB of 121.69 GB. The second run lowered the engine memory fraction from 0.4 to 0.2, and the kill came anyway. Section 2.3 shows the kill message, why it happens on a GB10, and what to change.

Choose the size

1. Start with the smallest model that can answer your task.
2. Multiply the parameter count by 16 bytes. On our 128 GB GB10, the limit fell between 27 GB and 64 GB of this estimate. Use half of your GPU memory as a first guess.
3. Run 4 steps at your real answer length. Read the node memory with free -g on the host, or in the Ray kill line. On a GB10, nvidia-smi reports no memory size.
4. Move up one size only when the short run leaves room.

To train a model of 4B or more, you need more GPUs (Part 6) or a recipe that keeps less optimizer state.

Train a larger model with LoRA

LoRA: low-rank adaptation. The trainer keeps the model weights frozen and trains two small matrices next to some of the layers. Only those small matrices get gradients and optimizer state.

LoRA is the usual way to train a model whose full trainer state does not fit. In the Miles version we ran, only the Megatron trainer supports it. The FSDP trainer of this manual refuses LoRA at start. To try it:

1. Switch to the Megatron trainer, as section 6.1 shows.
2. Add --lora-rank 16, or another rank above 0. The default rank of 0 turns LoRA off.
3. Keep one adapter. Under --colocate, Miles trains one adapter, and several adapters need a split layout.

The engine serves the same adapter that the trainer updates. We have not run LoRA or the Megatron trainer on our GB10. So we do not know if a 4B model fits this way on 128 GB.

When you finish this section, you can decide which memory each side gives back while the other side runs. You can also check an offload change without the heat pauses fooling you.

Our result: keeping the engine loaded saved 0.6 to 1.8 s of a rollout of about a minute. An earlier pair showed 12 percent, but shorter answers caused that gap.

Know the two switches

--colocate turns on offload for both sides. Two flags turn each side off:

• --no-offload-rollout keeps the engine weights and its KV cache on the GPU while the trainer works. The engine starts without its memory saver.
• --no-offload-train keeps the trainer weights and its optimizer state on the GPU while the engine works.

--offload-rollout-level kv_cache gives back the KV cache and the CUDA graph memory, and keeps the engine weights.

Add a switch to the run of section 1.1 with MILES_EXTRA_ARGS:

-e MILES_EXTRA_ARGS="--no-offload-rollout --sglang-mem-fraction-static 0.25"

The trainer side has its own timers. perf/sleep_time is the time the trainer takes to move its state to host memory after training, and Miles logs it with the next step. perf/wake_up_time is the time it takes to move the state back before the log probabilities. The engine release before training has no timer. The engine resume after the weight update is in perf/finalize_and_resume_engines_time.

Compare one change at a time, on the same steps

We ran a same-day pair: seed 1, 12 steps, the engine memory fraction at 0.25 in both runs. The only difference was --no-offload-rollout.

The table counts clean steps, as the front matter defines them.

The two medians cover different steps, and the rollout time follows the answer length. Early steps write answers of about 800 tokens and later steps about 690. So compare the steps that were clean in both runs:

We measured a small gain on steps 0, 1, and 8: 69.4 against 68.0 s, 58.5 against 56.7 s, and 55.8 against 55.2 s. The gain is 1 to 3 percent. No training phase escaped a pause in either run, so the training phase has no clean comparison. Its medians over all 12 steps were 240.3 and 239.1 s, with the pauses inside them. Raw runs

Our first pair of runs showed a larger gain, 52.0 s against 59.1 s. That pair also changed the memory fraction, and its clean steps were different steps. The lab notes explain why it does not hold.

One seed is not enough to rank the two settings. A gain of one or two seconds is inside the step-to-step noise of our GB10 under heat pauses.

Decide

Keep the default offload settings. Turn the engine offload off only when you have memory to spare. With the fraction at 0.25, the engine keeps about a quarter of the GPU memory while the trainer runs. On a GB10, the trainer needs that same memory. Section 2.1 shows how little room a 4B model leaves.

Our Qwen3-4B run finished its first rollout, then stopped in the training phase of that step, 15 minutes after start. The log holds no CUDA out-of-memory error. Ray had stopped the trainer because the node was nearly full.

When you finish this section, you can recognize a Ray memory kill and lower the memory that the run needs.

Our result: Ray stopped our 4B trainer twice, with the node at 115.86 and then 119.18 of 121.69 GB. Both stops were real memory pressure.

Read the kill message

Search the log for OutOfMemoryError before you change anything. A Ray memory kill names the node, not a GPU:

(TrainerController pid=2656) ray.exceptions.OutOfMemoryError: 1 worker(s) were killed due to the node running low on memory. Memory on the node (IP: 172.17.0.3, ID: 08904870a17c5413c623abfc21db2e7fcfdc39d5be90e997b2391f37) was 115.86GB / 121.69GB (0.952127)
(TrainerController pid=2656) Top 10 memory users: PID	MEM(GB)	COMMAND
(TrainerController pid=2656) 2657	36.77	ray::FSDPTrainRayActor.train
(TrainerController pid=2656) 3402	2.73	sglang::scheduler
(TrainerController pid=2656) 2650	1.33	ray::RolloutExecutor
To adjust the kill threshold, set the environment variable `RAY_memory_usage_threshold` when starting Ray. To disable worker killing, set the environment variable `RAY_memory_monitor_refresh_ms` to zero.

The node held 115.86 GB of 121.69 GB, which is 95.2 percent. The trainer process alone held 36.77 GB. The rest of the used memory was not in the process list.

Why this happens on a GB10

On a GB10, the CPU and the GPU use one pool of memory. Ray reads the memory of the whole node. So the GPU allocations count in the number that Ray watches. Those are the model weights, the optimizer state, and the KV cache. That is our explanation of the memory that the process list does not show. We did not measure the split.

That number is real. When it reaches 100 percent, the machine has no memory left for anyone. So a Ray kill at 95 percent is a warning that the run does not fit, and not a false alarm.

Lower the memory the run needs

Change the run, not the watcher:

1. Use a smaller model. On our GB10, only a smaller model got past the kill.
2. Lower the work per step. Use fewer prompts, fewer answers per prompt, or shorter answers. We did not test this on 4B.
3. Lower the engine memory. --sglang-mem-fraction-static sets the share of GPU memory that the rollout engine takes. In our 4B test, 0.2 in place of 0.4 did not prevent the kill.

The log also names RAY_memory_usage_threshold. Do not raise it on unified memory. A higher threshold lets the node get closer to full. If the node does run out, the operating system can stop any process on the machine, and that stop leaves no Ray message to read.

We ran the same Qwen3-4B job again with --sglang-mem-fraction-static 0.2 and the default Ray threshold. Ray stopped it again, at 119.18 GB. The trainer process held 36.75 GB, almost the same as in the first run, so the smaller engine share did not make room. Section 2.1 shows how to choose a model size that fits.

After each training phase, the trainer copies its new weights into the rollout engine. On one GPU this copy stays inside the GPU, and it takes under a second for the models we ran.

When you finish this section, you can measure the weight update and decide if it matters for your run.

Our result: the weight update took 0.64 s for Qwen3-0.6B and 0.91 s for Qwen3-1.7B, under one percent of a step.

Read the update time

Every step logs perf/update_weights_time in seconds. Read it next to perf/step_time. We ran 4 short steps for two model sizes, with answers cut at 256 tokens:

Steps 0 and 1 are warm-up steps. In these 4-step runs, their trainer offload took up to 4.1 s for 0.6B and 11.8 s for 1.7B. The table leaves them out.

Trainer offload: the time the trainer takes to move its weights and optimizer state to host memory after training, logged as perf/sleep_time. Trainer onload: the time it takes to move them back before the next log probabilities, logged as perf/wake_up_time.

The larger model is about 2.8 times the size, and its update took about 1.4 times as long. The trainer offload and onload grew faster than the update did. A likely reason is that the update sends only the weights, while the offload also moves the gradients and the optimizer state.

The short answers cut almost every reply, so the reward in these runs stayed near 0. These runs measure the copy, not the training.

Compare it with the step

In our 48-step run with Qwen3-0.6B, a full step took 152.6 s at the median, and the update took 0.64 s. That is under half a percent of the step. The 1.7B model gave a similar share.

So tune the rollout and the training phase first. Come back to the update only when it passes a few percent of the step.

Change the bucket size only with a reason

The trainer sends the weights in buckets. --update-weight-buffer-size sets the bucket size in bytes. Our runs used 536870912, which is 512 MiB.

A larger bucket means fewer transfers and a higher memory peak during the update. On one GPU, with updates under a second, we left it at 512 MiB.

When you finish this section, you can train a small model on GSM8K and choose how many answers to sample for each prompt.

Our result: in 24 steps, 4, 8, and 16 answers for each prompt gave reward gains inside one range. More answers made each rollout longer.

Run the recipe

The launcher from section 1.1 is this recipe. These flags shape each step:

--rollout-batch-size 32 --n-samples-per-prompt 8 --global-batch-size 256 --rm-type math

Group: the answers that the engine writes for one prompt. GRPO compares each answer with the other answers in its group, so a group where every answer gets the same reward teaches nothing.

Keep --global-batch-size equal to the prompts times the answers, here 32 times 8. Then each rollout gives exactly one optimizer step.

Choose the group size

We changed only --n-samples-per-prompt and set --global-batch-size to match. For 4 answers, the run of section 1.1 gets this line:

-e MILES_NUM_ROLLOUT=24 -e MILES_EXTRA_ARGS="--n-samples-per-prompt 4 --global-batch-size 128"

The table compares the first 24 steps of each run. The gain is the mean reward of steps 16 to 23 minus the mean of steps 0 to 7.

We measured no clear effect of the group size on the reward in 24 steps. The 16-answer run gained +0.097, and one of the three 8-answer seeds gained +0.098. The seed spread at 8 answers, 0.038 to 0.098, covers all the runs. Raw runs

The cost is clear. More answers mean more tokens to write in each step. The 4-answer rollout took 32.5 s on its 14 clean steps. Our 48-step run with 8 answers and the default seed ran on another day, before we added the heat guard, so no pause touched it. On the same 14 step numbers it took 59.1 s.

Every step of the 16-answer run overlapped a heat pause, so it has no clean time. Its median of 188.2 s includes the pauses.

The 16-answer run also logged 22,357 503 Service Unavailable lines, in bursts of up to 60 s. The client retried each request, and the run went on. With 512 open requests, the router probably found no free engine for a short time. We did not confirm that cause. Section 1.2 replays these bursts through the watchdog.

Read this result with its limits. The runs with 4 and 16 answers have one seed each. The 16-answer run finished all 24 steps. Then the time limit of our run queue stopped the container while it shut down, as the lab notes explain.

What we did not run

We did not run the rollout batch size sweep with 16 and 64 prompts for each step. We stopped it to keep the lab inside its time limit.

In a plain GRPO step, about half of our groups had every answer right or every answer wrong. Those groups give no learning signal. The filter drops them and asks the engine for more prompts until the step has enough groups that do.

When you finish this section, you can turn on the zero-variance filter and check how many groups it drops. You can also measure what the extra prompts cost.

Our result: the filter dropped 16 to 48 groups in each step, and most of them had every answer right. The rollout took 1.5 to 2.8 times as long. In 12 steps on one seed, the reward counted before the filter read higher than the run without it. We read that as a bias of the metric, not as learning.

Turn it on

Add two flags to the run of section 1.1 with MILES_EXTRA_ARGS:

-e MILES_EXTRA_ARGS="--dynamic-sampling-filter-path miles.rollout.filter_hub.common_filters.apply_reward_nonzero_std_filter --over-sampling-batch-size 48"

Our lab runs used an older path for the same filter, which Miles marks as deprecated. At the version we ran, the older path calls the filter shown here, so the results are the same.

Over-sampling batch: the number of prompts that the rollout sends to the engine at a time. It must be at least --rollout-batch-size.

The rollout sends 48 prompts. As each group finishes, the filter keeps it or drops it. When the kept and the waiting groups cannot reach 32, the rollout sends 48 more prompts. When 32 groups are kept, it stops the rest of the requests. The answers of the stopped requests are not used.

Count the dropped groups

The rollout writes three numbers for the filter in its perf N line:

Without the filter, the same groups stay in the batch. Read rollout/zero_std/count_0 and rollout/zero_std/count_1 to see how many there are.

We measured these counts over 12 steps with seed 1, with and without the filter: Raw runs

Without the filter, half of the groups that the trainer used taught nothing. With the filter, the step dropped 16 to 48 groups, with a median of 36. Most dropped groups were easy prompts that the model already answered right every time.

Measure the over-sampling cost

Every rollout of the filter run overlapped a heat-guard pause, so this run has no clean step. To compare the rollout times, we took the pause time out of each rollout. The run without the filter is our 12-step run with seed 1. With the pause time taken out, it gave a median of 58.0 s, close to the 59.1 s of its 6 clean steps.

Three steps dropped 16 groups or fewer, so one batch of 48 prompts was enough. Their rollouts took a median of 88.7 s, 1.5 times the 58.0 s without the filter. The other nine steps needed a second batch and took a median of 164.4 s, 2.8 times as long.

One batch of 48 prompts is 1.5 times the work of 32 prompts, which matches the first ratio. A second batch adds more, and the step must wait for the slowest groups that it keeps. The sum of the 12 step times was 113 minutes with the filter and 60 minutes without it, with the heat pauses in both.

Compare the reward

The filter changes what rollout/raw_reward means, because the dropped groups are not in it. To compare the two runs, read rollout/raw_reward_unfiltered for the filter run.

The mean over 12 steps was 0.686 with the filter and 0.657 without it. Our five baseline runs gave 12-step means of 0.638 to 0.660, so 0.686 is above all of them. The mean of the first four steps was also higher, 0.672 against 0.625, before the filter could have changed the model much.

We think the reason is in what the number counts. The reward before the filter counts only the groups that finished before the rollout stopped. The stopped requests are the slowest, with the longest answers. An answer cut at the length limit usually scores 0, as section 4.2 shows. So this number can lean high, and we read the higher mean as that bias, not as a learning effect.

This is one seed of 12 steps. The filter run finished all 12 steps. Then the time limit of our run queue stopped the container while it shut down. To see a reward effect, run more steps and a second seed, and compare against the spread of your baseline seeds.

Decide

Use the filter when many groups score the same for every answer, and when you can pay for a longer rollout. Set the over-sampling batch from your drop rate. With half of the groups dropped, a batch of 48 prompts for 32 kept groups was too small for most of our steps.

A math rollout is one prompt and one answer. An agent rollout is a conversation: the model writes, calls a tool, reads the result, and writes again. Each turn adds tokens that the trainer must score exactly as the engine produced them.

When you finish this section, you can pick a starting point for an agent recipe and know the settings that keep its turns consistent.

We have not run an agent recipe on our GB10. This page comes from the Miles source and examples.

Start from an example

Miles has agent recipes. Two of them are in the experimental folder. Pick the one closest to your task, and change its tools and data:

retool_v2 also has a mode that runs on one GPU with --fully-async --num-gpus-per-node 1. That is the closest start for a single-GPU lab.

Choose the generate function

The rollout of each sample comes from a generate function. Miles has three built in: one turn, many turns, and agentic tool calls. Name your own with --custom-generate-function-path, which takes a Python import path.

A generate function receives one sample, its sampling settings, and the router address. It fills in the sample: the tokens, the log probabilities from the engine, and a loss mask. The loss mask marks the tokens that the model wrote, and only those tokens count in the loss. Tool output is context, not model output.

Keep the tokens of each turn exact

Between turns, an agent loop often turns tokens into text and back into tokens. That round trip can change the tokens, and then the trainer scores a sequence the engine never produced.

Miles avoids that with a session server that passes tokens in and tokens out for every turn. With --use-session-server, the router also keeps all turns of one conversation on the same engine. So each turn reuses the cache of the turns before it.

Partial rollout from section 4.2 does not work with these recipes. The session server and the multi-turn and agent generate functions refuse it.

What to watch

• The share of tokens that the model wrote against the share that tools wrote. A recipe where tools write most of the tokens teaches the model little per step.
• The gap between the trainer and the engine, as in section 5.2. A gap far above your single-turn range points to a token mismatch between turns.
• The step time. Tools add wall-clock time that the GPU spends waiting.

In a mixture-of-experts model, a router picks a few experts for each token. The engine and the trainer each run that router. A tiny numeric difference can make them pick different experts for the same token, and then they compute that token with different weights.

When you finish this section, you can decide when to turn on routing replay and what to watch when you do.

We have not run a mixture-of-experts model with Miles on our GB10. This page comes from the Miles source and release notes.

Why the gap is worse on a MoE model

On a dense model, the gap between the two sides comes from small rounding differences. On a MoE model, the same rounding can flip an expert choice. A flipped choice is a large difference. The token goes through other weights, and its probability can change a lot.

So a MoE run can show a larger gap than a dense model, even when the dense parts of the model agree well.

Turn on routing replay

The flag works only for MoE models. Add it with MILES_EXTRA_ARGS, as in section 1.1. The Qwen MoE models are too large for full training on one 128 GB machine, as section 2.1 shows. Our launcher starts one GPU, so use a launcher sized for your GPUs, and pass the same flag:

-e MILES_EXTRA_ARGS="--use-rollout-routing-replay"

Routing replay: the engine records the experts it chose for each token during rollout, and the trainer uses those same experts in its forward pass. The flag works only for MoE models.

Some models need a matching engine setting. The Miles notes for GLM-5.3-Flash ask for --sglang-moe-runner-backend triton with replay.

What the Miles team reports

The Miles team ran colocated MoE jobs with routing replay on 8 nodes of 4 GB300 GPUs. Their v0.1.1 release notes give the gap per token for each model. The sources list the notes and the routing replay paper.

The GLM-5.3-Flash page of the Miles documentation reports a longer run on 16 nodes of 4 GB300 GPUs. Its gap was 0.0068 to 0.0106 over the first 11 steps.

These are short runs on hardware we do not have. Read them as the range a working setup reaches, not as a promise for your model.

A known limit

If the engine pauses a request and later prefills it again under new weights, the routes from the second prefill can overwrite the recorded routes. An open Miles issue tracks this. If you use a pause mode that retracts requests, check the gap after each weight update.

The reward decides what the model learns. Miles has a set of built-in rewards, and you can name your own function when none of them fits your task.

When you finish this section, you can plug in your own reward function and check its advantages before a long run.

We have not measured a custom reward on our GB10. Our runs used the built-in math reward, which gives 1 for a correct answer and 0 otherwise.

Check the built-in rewards first

--rm-type picks a built-in reward. The ones you are most likely to need:

Add the prefix boxed_ to a math type, such as boxed_math, to grade only the answer inside \boxed{}.

Plug in your own function

Save the function as ~/miles-lab/my_rewards.py on the host. The container sees it as /root/lab/my_rewards.py:

import re


def last_number(text):
    found = re.findall(r"-?\d+(?:\.\d+)?(?:[eE][-+]?\d+)?", text.replace(",", ""))
    return float(found[-1]) if found else None


async def custom_rm(args, sample, **kwargs):
    answer = last_number(sample.response)
    label = last_number(str(sample.label))
    return 1.0 if answer is not None and answer == label else 0.0

This example grades a number answer. It compares the last number in the answer with the label as a value. So an answer of 142 does not match a label of 42. Change the extraction for answers that are not numbers.

Then add two settings to the run of section 1.1. PYTHONPATH lets Miles import the file from the lab folder, and the flag names the function by its import path:

-e PYTHONPATH=/root/lab -e MILES_EXTRA_ARGS="--custom-rm-path my_rewards.custom_rm"

The launcher passes PYTHONPATH on to the Ray workers that score the answers. When the import works, the first rollout 0 line shows a rollout/raw_reward that your function computed.

If Miles cannot find the file, the log shows ModuleNotFoundError: No module named 'my_rewards'. The run does not stop by itself after this error. In our test the job stayed up for two hours with no rollout step. The watchdog did not stop it, because the engine wrote only 44 lines of 503 Service Unavailable in that time. When you see the error, stop the container, fix PYTHONPATH, and start the run again.

Write the function with async def. The flag help shows a plain function, but Miles awaits the result, so a plain function fails. The function receives the run arguments and one sample, with the answer in sample.response and the label in sample.label. If it returns a dict, name the field that holds the reward with --reward-key. With --group-rm, Miles passes the whole group as a list of samples instead, and the function must return one reward for each sample.

Test it before the run

Call the function inside the container before the run. This command scores one answer:

docker run --rm -v "$HOME/miles-lab:/root/lab" -e PYTHONPATH=/root/lab radixark/miles:latest python -c "import asyncio, types, my_rewards; s = types.SimpleNamespace(response='So the answer is 42.', label='42'); print(asyncio.run(my_rewards.custom_rm(None, s)))"

It should print 1.0. Then run the function on ten answers that you have scored by hand. Include a correct answer, a wrong answer, an empty answer, and an answer cut at the length limit. A reward that crashes on an empty answer can stop the rollout of the whole step.

Check the advantages of fractional rewards

GRPO subtracts the mean reward of each group from each answer, then divides by the standard deviation of the group. When every answer in a group gets the same reward, every advantage should be 0, and the group teaches nothing.

A report in the Miles issue tracker shows that this fails for some fractions. In float32, the mean of 16 rewards of 0.3 is not exactly 0.3, so a tiny remainder stays. The division by a standard deviation near 0 makes it large: about 0.029 per answer. A binary reward of 0 or 1 does not hit this, because its mean is exact.

If your reward gives fractions, find groups where every answer has the same reward and check that their advantages are 0. If they are not, round the rewards to values that a float32 holds exactly, such as multiples of 0.25.

In a normal Miles step, rollout runs, then training runs, then rollout runs again. On many GPUs, the rollout GPUs wait while the trainer works, and the trainer waits while the rollout GPUs work. Async training removes part of that wait, and pays for it with samples from older weights.

When you finish this section, you can decide if async training helps your layout, and which settings control the cost.

We have not run async training on our GB10. This page comes from the Miles source and documentation.

Two async modes

Both modes start from train_async.py instead of train.py:

In the first mode, with the default update interval, every sample is at most one weight version old. In the fully async mode, the age of a sample depends on how fast the trainer drains the buffer.

What async gives on one GPU

With --colocate, the trainer and the engine share the GPU. Miles allows colocation only with --fully-async, so the one-step-ahead mode needs separate GPUs. The fully async mode can run on one GPU with --colocate, but only with the Megatron trainer, not the FSDP trainer of this manual. Generation also stops for the whole training window, so the two phases still take turns.

So on one GPU, async training does not save time. It adds the cost of older samples and gives no overlap. Use it on one GPU only to learn its settings before a larger run.

Control the age of the samples

On a layout with separate rollout GPUs, these settings decide how old a sample can get:

• --update-weights-interval sets how many steps run between weight updates to the engine.
• --max-weight-staleness drops samples that are older than this many weight versions. Unset means no limit.
• --async-unused-samples-handler decides what happens to a dropped sample: drop or retry.
• --async-max-concurrent-samples caps the number of samples that the engines generate at one time.
• --pause-generation-mode decides what happens to running requests during a weight update: abort, retract, or in_place. The default is retract. The fully async mode refuses abort, and refuses in_place with --colocate.

--over-sampling-batch-size has no effect under --fully-async.

Lower the learning rate as samples get older

Sample age and learning rate work together. A 2026 study of async GRPO on Llama models of 1B and 3B parameters changed both. The sources list it as the study of collapse in asynchronous GRPO. The largest stable learning rate fell in proportion to the number of steps between weight updates. Twice the steps allowed about half the learning rate.

We have not tested this. Use it as a starting rule: when you raise --update-weights-interval or the age limit, lower the learning rate by the same factor. Then watch the gap and the reward.

What to watch

• The weight version of each sample. The rollout metrics report how old the samples are.
• The share of dropped or retried samples. A high share means the engine works for nothing.
• The gap between the trainer and the engine, as in section 5.2. Older samples add a gap that the kernels did not cause.

On our single GB10, we cannot measure the overlap that async training gives, because there is no second GPU to overlap with.

A rollout step ends when its last answer ends. If a few answers run to the length limit, the whole step waits for them, and the engine has idle capacity while it does.

When you finish this section, you can measure how much the long answers cost you, and keep them from slowing every step.

We have not run partial rollout on our GB10. The measured part of this page is the cut share of our baseline runs.

Measure the long answers first

rollout/truncated reports the share of answers that hit the length limit, --rollout-max-response-len. Our three baseline seeds used a limit of 1,024 tokens. In their first 8 steps, 0.30 to 0.34 of the answers hit the limit. In their last 8 steps, the model had learned shorter answers, and 0.08 to 0.12 did. Section 5.3 has the numbers for each seed.

A high cut share has two costs. A cut answer usually stops before its final answer, so the math reward scores it 0. And the step waits for the longest answers.

Choose between a higher limit and partial rollout

If the cut answers are close to right, raise --rollout-max-response-len. The model then gets credit for long correct answers, and each step takes longer.

If a few very long answers set the step time, use partial rollout instead:

-e MILES_EXTRA_ARGS="--partial-rollout --over-sampling-batch-size 48 --dynamic-sampling-filter-path miles.rollout.filter_hub.common_filters.apply_reward_nonzero_std_filter"

The filter is the one from section 3.2.

With partial rollout, the step takes the groups that finish first. Groups that are not done go back to the data buffer, and a later step continues them from the tokens they already have. The engine does not generate those tokens again.

Partial rollout works together with over-sampling. --over-sampling-batch-size sets the size of each new batch of prompts that the rollout starts when it runs short of finished groups. Keep such a run synchronous. --fully-async and the session server both refuse --partial-rollout. The multi-turn and agent generate functions do not support it either.

The cost of continued answers

A continued answer starts under one set of weights and ends under a later one. So part of its tokens come from older weights. Watch the gap between the trainer and the engine, as in section 5.2, after you turn partial rollout on.

Speculative decoding, a lever we have not measured

Speculative decoding is another way to shorten rollout. A cheap draft guesses the next few tokens, and the engine checks all of them in one forward pass. Miles passes the SGLang speculative flags through with the --sglang- prefix. For a model with its own MTP layers, such as GLM-4.7 or DeepSeek-V3, the Miles documentation gives these flags:

--sglang-speculative-algorithm EAGLE --sglang-speculative-num-steps 3 --sglang-speculative-eagle-topk 1 --sglang-speculative-num-draft-tokens 4

There are two kinds of draft to start from:

• An MTP draft. The flags above use the MTP layers of the model as the draft. A model without them needs a separate draft, set with --sglang-speculative-draft-model-path. As training changes the policy, the draft falls behind, and fewer of its tokens pass the check. Miles can train the MTP layers with the policy: --mtp-num-layers 1 --enable-mtp-training, on the Megatron trainer only.
• An n-gram draft. --sglang-speculative-algorithm NGRAM needs no draft model. It guesses from tokens that the engine already generated. SGLang supports it on CUDA only, and it turns off the overlap scheduler.

The Miles documentation advises against speculative decoding on dense models below about 13B parameters when rollout is the bottleneck. There, the check can cost more than the draft saves. Our Qwen3-0.6B is in that range, and we have not measured it. If you try it, compare the rollout time and the gap over the same steps and seed.

Rollout is often the longest phase of a step. Running the engine in FP8 halves the bytes of each weight that the engine reads. The trainer keeps its weights in BF16, so the two sides now compute with different numbers on purpose.

When you finish this section, you can choose a low-precision mode, and know which correction to pair with it.

We have not run FP8 rollout on our GB10. This page comes from the Miles low-precision documentation and arguments.

Pick a mode

The Miles documentation lists four ways to mix precisions:

Start with mode 1. It changes only the checkpoint that the engine loads.

Mode 1 is the easiest start, but it is not safe in every case. We have not tested these limits. They come from published reports:

• A 2026 study found that FP8 rollout with BF16 training matched BF16 on rollouts below about 4K tokens. At 8K tokens it drifted away, and at 16K tokens it collapsed. It also failed sooner on tasks that were hard for the base model.
• A post by the Miles team and its partners found that this mismatch grows with MoE model size, from 30B to 1T parameters. Their answer is FP8 on both sides, as in mode 2.
• Another RL library warns about small models. There, the cost to quantize the weights at each weight update can be larger than the time that FP8 saves.

The sources list the study, the post, and the library documentation.

Set up mode 1

Give the engine an FP8 checkpoint and give the trainer the BF16 weights:

• --hf-checkpoint points at the FP8 checkpoint. The engine loads it.
• The trainer loads the BF16 weights. On the Megatron trainer, --ref-load points at them.

Mode 1 needs the Megatron trainer. The FSDP trainer loads its own weights from --hf-checkpoint too, so with FSDP both sides would load the FP8 checkpoint. Our GB10 runs use FSDP, which is one reason we have not run mode 1.

Block-wise FP8: a format that stores one scale for each block of 128 by 128 weights. The documentation lists it for Hopper GPUs, for Blackwell B200 and newer, and for AMD MI350X and MI355X.

Check the first weight update with --check-weight-update-equal --check-weight-update-allow-quant-error. The second flag lets the check allow the rounding that FP8 adds.

Pair it with a correction

FP8 rounding adds to the gap between the trainer and the engine. The gap now has a cause that no kernel setting removes. The documentation pairs mode 1 with routing replay on MoE models, and with a capped importance-sampling correction as an option:

• Importance sampling with a cap: --use-tis --tis-clip 2.0 weights each token by the ratio of the trainer probability to the engine probability, and caps that ratio.
• Routing replay: on a MoE model, add --use-rollout-routing-replay, so that FP8 rounding does not also change the expert choices. Section 3.4 explains it.

Section 8.2 compares the corrections.

What to measure

Measure the gap at step 0 and over the first steps, with and without FP8 rollout. Measure the rollout time too, because we have no numbers for how much FP8 saves on a small model.

The documentation lists B200, B300, GB200, and GB300 for modes 3 and 4. Our GB10 is not on that list, and it is not on the block-wise FP8 list either, so we cannot test these modes as documented.

The engine samples each token from a cut-down set of the vocabulary when you use top-p or top-k. The trainer scores the same token over the full vocabulary. So the two sides give different probabilities for reasons that have nothing to do with the kernels.

When you finish this section, you can turn on sampling replay and check that it lowers the gap.

Our result: top-p and top-k replay lowered the median gap from 0.0153 to 0.0135, 11.5 percent, on one seed.

Why the engine number is higher

Top-p keeps the smallest set of tokens whose probabilities add up to at least the top-p value. The engine then scales the kept probabilities so that they add up to 1 again. Take two sides that compute the same scores before the cut. For a kept token, the engine log probability is then higher than the full-vocabulary score by minus the log of the kept mass. If the kept tokens hold 0.9 of the mass, the difference is about 0.105. Top-k cuts the set in the same way, and the kernel gap comes on top of this.

That difference is not noise. It is different for each token, and it always pushes the ratio between the trainer and the engine the same way. A 2026 issue report for another RL library describes this bias in its importance-sampling ratio. Sampling replay removes it, because the trainer then scores over the same kept set.

Turn it on

Add the sampler settings to the run of section 1.1 with MILES_EXTRA_ARGS:

-e MILES_EXTRA_ARGS="--rollout-top-p 0.95 --rollout-top-k 64"

Miles turns on sampling replay by itself when --rollout-top-p is below 1 or --rollout-top-k is above 0. The engine then returns the set of tokens that each step could sample from. The trainer scores each token over that set only.

Set a positive top-k whenever you set a top-p below 1, because Miles requires both for replay. Miles refuses to start when replay is on together with a KL loss or on-policy distillation. It also refuses replay with --recompute-logprobs-via-prefill.

Check the gap

Read train/train_rollout_logprob_abs_diff in each step. It is the mean absolute difference between the log probabilities of the trainer and of the engine.

run                    attention_backend  enable_deterministic_inference  sampling_backend  --rollout-top-p  --rollout-top-k
e6a-base               flashinfer         False                           flashinfer        not set          not set
e11-topp-topk          flashinfer         False                           flashinfer        0.95             64

We measured a gap 11.5 percent lower with replay. No step of the replay run reached the lowest step of the run without it. The reward stayed in the same range. Raw runs

This is one seed of 12 steps on Qwen3-0.6B. The step ranges did not overlap, so we report the direction. Run a second seed before you report the size for your own model.

What replay does not fix

Replay removes the part of the gap that comes from the sampler. The rest most likely comes from the kernels, because the engine and the trainer add the same numbers in a different order. Section 4.5 shows that deterministic inference does not remove that part. Section 8.3 explains what does.

We turned on deterministic inference to close the gap between the trainer and the engine. The gap did not move. What changed was that two runs with the same seed now gave the same rollout at step 0.

When you finish this section, you can decide when deterministic inference is worth its cost.

Our result: with and without deterministic inference, the median gap stayed between 0.0151 and 0.0153. Two deterministic runs repeated the rollout of step 0 exactly.

Turn it on

Add the engine flag and an attention backend that supports it to the run of section 1.1, both through MILES_EXTRA_ARGS:

-e MILES_EXTRA_ARGS="--sglang-enable-deterministic-inference --sglang-attention-backend triton"

The launcher also reads MILES_SGLANG_ATTENTION, but only in true on-policy mode with MILES_TRUE_ON_POLICY=1. In a normal run, that variable changes nothing.

On our GB10, both triton and flashinfer started with the flag. The engine also switches its sampler, and the log says so:

[2026-10-21 13:45:36] Sampling backend is set to pytorch for deterministic inference.
[2026-10-21 14:55:18] Sampling backend is set to pytorch for deterministic inference.

The gap stays the same

We ran four 12-step runs with seed 1 on Qwen3-0.6B:

We measured the same gap with and without deterministic inference. The four medians are within 0.0002 of each other. Raw runs

This points to the kernels as the cause: the two sides compute with different kernels. Deterministic inference makes the engine give the same result for the same input, whatever the batch around it. It does not make the engine compute like the trainer.

The first step repeats

The two Triton runs gave the same gap at step 0, to six decimal places: 0.015404. From step 1 they drifted apart.

The engine repeated itself, and the trainer most likely did not. These runs did not set the trainer's deterministic mode, so the weights after the first update can differ between the runs. We have not tested that with the trainer flags on.

So use deterministic inference when you need to repeat an experiment, for example to test a reward function on the same samples. Do not use it to lower the gap.

The cost

Deterministic kernels are slower. Four steps had no heat-guard pause in all four runs: steps 0, 1, 4, and 10. On those four steps, the median rollout took 56.5 s with the default engine. The 59.1 s of section 5.3 is the same default run over its 6 clean steps. It took 79.1 to 82.0 s in the three deterministic runs, with answers of about the same length.

That is 40 to 45 percent more rollout time, from one run of each setting on a GPU under heat pauses. Treat it as an unconfirmed result. Measure the cost on your own GPU before you leave the flag on for a long run.

True on-policy mode promises a gap of exactly 0 between the trainer and the engine. On our GB10 it did not start, and the run did not tell us so.

When you finish this section, you can check your GPU and model before you plan a true on-policy run.

Check the contract before you start

The Miles versions we checked have one true on-policy contract. It covers dense Qwen3 models: Qwen3-0.6B and three Qwen3-4B variants. Both sides must run FlashAttention 3: the engine as its attention backend, and the trainer as its attention implementation.

FlashAttention 3 is built for Hopper GPUs. The engine check accepts compute capability 8.0 to 9.0 and refuses newer GPUs. So check two things before you plan the run:

1. Your GPU is a Hopper GPU, such as an H100 or an H200.
2. Your model is in the contract.

If either answer is no, true on-policy mode cannot start.

What happens on another GPU

Our GB10 is a Blackwell GPU with compute capability 12.1. The engine stopped during startup:

[2026-10-17 21:03:03] Enable deterministic inference because of legacy rl_on_policy_target.
AssertionError: FlashAttention v3 Backend requires SM>=80 and SM<=90. Please use `--attention-backend flashinfer`.

The trainer kept going and sent requests to the dead engine for an hour. Section 1.2 shows that part, and the watchdog that stops it.

The error suggests FlashInfer. Do not take that advice for true on-policy mode, because the contract names FlashAttention 3 on both sides. With FlashInfer you get a normal run, not a bit-exact one.

What to use instead

Outside the contract, watch the gap and lower it where you can:

• Section 4.4 lowers the part that comes from top-p and top-k sampling.
• Section 4.5 makes the engine repeat itself, which helps you repeat an experiment.
• Section 5.2 checks that the gap at step 0 is in a healthy range.

On our GB10, every step of every finished Qwen3-0.6B run without top-p or top-k gave a gap between 0.0136 and 0.0164. That is the range to compare against on this setup.

The reward is the signal everyone reads first, and it is the slowest to tell you that something is wrong. Seven other signals show a broken run earlier.

When you finish this section, you can decide if a run is healthy from its logged signals. You can also tell which changes need action.

The eight signals

These are the ranges over all 48 steps of our three baseline seeds. The runs trained Qwen3-0.6B on GSM8K, with 32 prompts of 8 answers each and a limit of 1,024 tokens.

Clip fraction: the share of tokens where the update hit the clip limit. Our runs did one optimizer update per rollout. So the policy of the trainer stayed the one that scored the samples, and nothing was clipped.

Our runs logged train/entropy_loss as 0 at every step, so this setup does not show the entropy of the model.

Read trends, not steps

From one step to the next, the reward moved by up to 0.21 in our runs. A single step tells you little. Read the mean over five or more steps.

All three seeds showed the same trend: the reward rose and the answers got shorter. Section 5.3 gives the numbers for each seed.

Send the signals to a dashboard

The log is enough for ten steps. For a long run, a dashboard is easier to read. Miles can write the same train/, rollout/, and perf/ metrics to Weights & Biases or to TensorBoard. We read every number in this manual from the log. We tested both trackers with a 2-step run on our GB10.

For Weights & Biases without an account or a network, write the run to a local folder:

-e MILES_EXTRA_ARGS="--use-wandb --wandb-mode offline --wandb-project miles-lab --wandb-group plain --wandb-dir /root/lab/runs/plain/wandb"

Keep --wandb-group. Without it, the run stops at start with TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'. The log should show W&B offline mode enabled. Data will be saved locally., and an offline-run- folder appears under ~/miles-lab/runs/plain/wandb/wandb/.

For an online run, leave out --wandb-mode offline. Keep the key out of the command line, because commands go to the shell history and the process list. Put WANDB_API_KEY=<your key> in a file that only you can read, such as ~/miles-lab/wandb.env with mode 600, and add --env-file ~/miles-lab/wandb.env to the docker run line. Do not put the key in MILES_EXTRA_ARGS, because the launcher prints the full command in the log.

For TensorBoard, give Miles a folder and turn the writer on:

-e TENSORBOARD_DIR=/root/lab/runs/plain/tb -e MILES_EXTRA_ARGS="--use-tensorboard"

The log should show Saving tensorboard log to /root/lab/runs/plain/tb. Then open the folder on the host with tensorboard --logdir ~/miles-lab/runs/plain/tb.

Act on these changes

Keep your own ranges. These numbers hold for our model, data, and machine. Run two or three seeds of your own setup, and use their ranges instead.

Step 0 is the cheapest test you get. No training has happened yet, so every difference between the trainer and the engine comes from the setup, not from learning.

When you finish this section, you can check the first step of a run and stop a broken run before it trains on wrong numbers.

Check two values at step 0

At step 0, the trainer computes the log probabilities of the samples with the same weights that the engine used. So two values must be at 0 in the step-0 metrics line:

• train/ppo_kl, the KL between the trainer's old and new probabilities. Before the first update they are the same.
• train/pg_clipfrac, the share of clipped tokens. Nothing can be clipped before the first update.

Miles asserts both in its own tests: train/ppo_kl below 1e-9 and train/pg_clipfrac below 1e-10. That assert runs only on the Megatron trainer with --ci-test, on the first optimizer step of each rollout. --ci-disable-kl-checker turns it off. On the FSDP trainer, or without that flag, read the values yourself.

All our runs logged a clip fraction of 0 at step 0.

Compare the gap with your range

The gap between the trainer and the engine is never 0 outside true on-policy mode, because the two sides use different kernels. At step 0 it must still sit inside the range of your healthy runs:

The gap depends on the model and the sampler, so build the range for your own setup. A step-0 gap several times your range means the two sides do not compute the same model. Common causes are a wrong checkpoint path on one side, or a model format that one side reads in a different way.

Check the weights directly

Add --check-weight-update-equal to a short test run, with -e MILES_NUM_ROLLOUT=2 -e MILES_EXTRA_ARGS="--check-weight-update-equal" on the run of section 1.1. After the first weight update, Miles compares the engine weights with the trainer weights. With FP8 or another low-precision engine, add --check-weight-update-allow-quant-error so that the check allows the expected rounding.

Run this check once for each new model or image, not on every long run.

We ran the same GRPO setup three times. The first run used the default seed, which the engine logged as 1234, and the others used seeds 2 and 3. Nothing else changed. The three runs still ended in different places.

When you finish this section, you can tell a real effect from seed noise, and time a run on a hot GPU.

Our result: in our 12-step run with seed 1, the median rollout was 83.9 s over all steps. Over its 6 clean steps, it was 59.1 s.

Know the noise of your setup

Each run trained Qwen3-0.6B on GSM8K for 48 steps, with 32 prompts and 8 answers each. We compare the mean of the first 8 steps with the mean of the last 8:

Length: the mean answer length in tokens. Cut share: the share of answers that reached the 1,024-token limit.

All three seeds learned: the reward rose and the answers got shorter. The final reward still spread from 0.736 to 0.766, so a difference under 0.03 in that number is inside the noise of this setup.

The gap between the trainer and the engine stayed between 0.0136 and 0.0164 in every step of the three runs.

Compare a change against the spread

Before you report that a change helped:

1. Run the change with at least two seeds.
2. Compare it with the spread of your baseline seeds, not with one baseline run.
3. Report the change only when it falls outside that spread.

A change of 0.01 in reward on one seed tells you nothing on this setup. A change of 0.05 on two seeds is worth a third run.

Time only clean steps

Our GB10 got hot under long runs. The heat guard of section 1.2 pauses the container at 84 C. It resumes the container after the GPU cools, or after two minutes. A pause adds to the phase time of the step it falls in. We expect a pause to leave the samples and the losses unchanged. We have not tested that yet.

So we report times from steps whose rollout overlapped no pause. Our summary script finds those steps from the pause log and the step timestamps. The lab notes give the rule.

Clean steps are not enough on their own. The same setting gave a median rollout of 47.5 s in our first 48-step run. That run came before we added the heat guard, so no pause could touch it. Our 12-step run with seed 1, four days later, gave 59.1 s over its 6 clean steps. On the same 6 step numbers, the 48-step run took 60.9 s.

The rollout time follows the answer length, and the 48-step median includes the late steps with short answers. So compare times on the same step numbers, and repeat them with a second seed.

On one GPU, the trainer and the engine take turns, and the weight update is a copy inside the GPU. On many GPUs you have a choice. Keep both sides on the same GPUs and take turns, or give each side its own GPUs and send the weights across.

When you finish this section, you can choose a placement and a weight transfer method for a run on many GPUs.

We cannot measure this on our single GB10. This page comes from the Miles source and documentation.

Choose a placement

Set the sizes with these flags:

• --actor-num-nodes and --actor-num-gpus-per-node set the trainer GPUs.
• --rollout-num-gpus sets the rollout GPUs in a split layout.
• --rollout-num-gpus-per-engine sets how many GPUs each rollout engine uses.

Start colocated. Split only when the rollout phase takes most of the step and you have the GPUs to give it. A split layout leaves trainer GPUs idle during rollout unless you also run async training, as in section 4.1.

Choose how the weights move

In a split layout, the weights leave the trainer GPUs after every update. --update-weight-transfer-mode picks the method:

Under --colocate, Miles uses a copy inside the GPU instead, and the transfer mode does not apply.

Choose the trainer

This manual uses the FSDP trainer, because it loads a Hugging Face model folder as it is. Miles has two other trainers, and --train-backend picks one:

Some features in this manual work only on Megatron: fully async training with --colocate, train fault tolerance, MTP training, and the step-0 checks of --ci-test. The Miles documentation recommends Megatron for large models and for jobs on many nodes.

torch_dist: the checkpoint format of Megatron. It does not depend on the parallel layout, so you can change the layout later without a new conversion.

Convert a model once before the first Megatron run. The architecture flags come from the model file of Miles for that model, here qwen3-4B:

cd /root/miles
MODEL_ARGS=$(python3 miles/utils/external_utils/model_args_utils.py qwen3-4B)
PYTHONPATH=/root/Megatron-LM python3 tools/convert_hf_to_torch_dist.py $MODEL_ARGS --hf-checkpoint /root/lab/models/Qwen3-4B --save /root/lab/models/Qwen3-4B_torch_dist

Run these lines inside the container, with the lab folder mounted as in section 1.1. Our launcher builds an FSDP run only. For Megatron, start from a launcher of Miles for your model, in the scripts folder of the Miles source. We have not run the Megatron or the torchtitan trainer on our GB10. We ran only the conversion to torch_dist for Qwen3-0.6B, and it wrote the checkpoint folder.

Start a run on more than one node

Every node must be in one Ray cluster before Miles submits the job. The launchers of Miles do it in one of two ways:

• One command for each node role. A worker command joins the cluster of the head node and waits. A train command starts the head node, waits for all GPUs, and submits the job.
• One command on the head node, which connects to the other hosts over ssh and adds them to the cluster.

To use a Ray cluster that you started yourself, set MILES_SCRIPT_EXTERNAL_RAY=1. The launcher then skips its own ray start and only submits the job. Set MASTER_ADDR to the address of the head node on every node, so that Ray and the trainer use the right network interface. Then set the trainer size with --actor-num-nodes and --actor-num-gpus-per-node.

Section 6.3 lists a known problem of some launchers with a Ray cluster of your own. We have not run Miles on more than one node.

What to measure after the split

• perf/update_weights_time against perf/step_time. On many machines the transfer is no longer a copy inside one GPU.
• The idle time on each side, from perf/train_wait_time and the rollout timers.
• The gap between the trainer and the engine, as in section 5.2, after the first update with the new method.

You want an eval score every few steps, such as accuracy on a held-out math set. That eval needs generation, and generation needs engines. The question is which engines.

When you finish this section, you can choose where evaluation runs, and set it up.

We cannot measure the dedicated fleet on our single GB10, because it needs GPUs of its own. This page comes from the Miles source and documentation.

Three places to run evaluation

You cannot select the fleet together with a checkpoint eval function for your own service. Rollout fault tolerance from section 6.3 also does not work with a fleet.

Evaluate on the training engines

Set the interval and the data:

-e MILES_EXTRA_ARGS="--eval-interval 10 --eval-prompt-data gsm8k_test /root/lab/datasets/gsm8k/test.parquet"

The launcher of section 1.1 already downloads the GSM8K test split into the lab folder.

Rollout stops while the eval runs, and requests that are already running finish first. The engines already hold the current weights, so no extra weight transfer happens. Use this mode on one GPU, and for small eval sets.

This command did not work on our GB10. We ran it twice on a 2-step run with --eval-interval 1. Both times the first eval sent the 1,319 test questions, the router answered with 503 Service Unavailable, and the job stopped before step 0. We did not find the cause. Test the eval on a 2-step run before you add it to a long run.

Evaluate on a dedicated fleet

--eval-num-gpus 1 --eval-interval 10 --eval-hf-dir /dev/shm/miles_eval_hf --eval-prompt-data aime /root/lab/datasets/aime/aime.jsonl

Put your own eval file in the lab folder first. The launcher does not download AIME. A fleet also needs a GPU of its own, so our one-GPU launcher cannot start it.

The fleet runs its own engines behind its own router. On each eval step, the trainer writes a checkpoint snapshot to --eval-hf-dir, hands it to the fleet, and keeps training. The fleet loads the snapshot. It does not take part in the training weight updates.

The eval point still appears at the step it belongs to, even when it finishes a few steps later. eval/lag_steps reports how many steps late it finished.

The fleet engines copy every --sglang-* setting of the rollout engines. Change one with the matching --eval-sglang-* flag, for example --eval-sglang-mem-fraction-static 0.9, because the fleet does not share its GPUs with training.

Choose by the cost of the pause

Start with the training engines. Move to a fleet when the eval pause becomes a large share of the step time, and when you can spare whole GPUs. A staging folder in /dev/shm keeps the snapshot off the disk.

A run on one GPU fails in simple ways, and you restart it. A run on hundreds of GPUs for days will lose an engine or a node along the way. The question is how much of the run you lose with it.

When you finish this section, you can set up checkpoints and fault tolerance for a long run, and test them before you rely on them.

We cannot test fault tolerance on our single GB10. On one GPU, a dead engine means a dead run, as section 1.2 shows. This page comes from the Miles source and release notes.

Save checkpoints first

Checkpoints limit the work you lose when you restart after a failure. They do not help when the storage fails, when a checkpoint is damaged, or for the steps after the last checkpoint. Set them before anything else.

The container starts with --rm, so a folder inside it goes away with it. Mount a host folder for the checkpoints, and add two lines to the run of section 1.1:

-v "$HOME/miles-lab/ckpt:/ckpt" -e MILES_EXTRA_ARGS="--save /ckpt/run-a --save-interval 20 --load /ckpt/run-a"

--save-interval counts steps. Choose it from the step time, so that a crash loses less work than you can accept. With a step of five minutes, an interval of 12 loses at most one hour.

Resume from a checkpoint

Start the same command again after a failure, with a new log file, such as > runs/plain-resume.log. The redirect of section 1.1 empties the old log, and you need it to find the cause of the failure. --load names the parent folder, never one iter_ folder inside it. Miles reads its tracker file to find the last saved step. On the first run, the folder is empty, and Miles starts fresh.

With the FSDP trainer of this manual, a resume restores these parts of the run:

• the model weights and the optimizer state
• the learning-rate schedule and the random state
• the step count, so the run continues at the next step
• the position in the prompt data, which the rollout side saves next to the trainer state

The log of a resumed run should show [FSDP] Loaded model from and [FSDP] Loaded optimizer from, each with the folder of the step. If these lines are missing, the run started fresh. Our first run with an empty folder printed no message about the missing checkpoint. In our resume check, the first trainer line after the load was step 2.

Test a restart on a short run before the long run starts. Miles always saves after the last step, so run the command above with MILES_NUM_ROLLOUT=2. Then run it again with MILES_NUM_ROLLOUT=4. The first step line of the second run should be step 2, not step 0. We ran this test on our GB10 with Qwen3-0.6B, and it passed. We also ran the converter below on the step 2 checkpoint, and it wrote a model folder with the weights and the tokenizer files.

Convert a checkpoint for serving

The FSDP trainer writes PyTorch distributed checkpoints, and other tools cannot load them as they are. Miles has a converter to a Hugging Face model folder. Run it inside the container:

python3 /root/miles/tools/convert_fsdp_to_hf.py --input-dir /ckpt/run-a/iter_0000020 --output-dir /root/lab/models/run-a-step-20 --origin-hf-dir /root/lab/models/Qwen3-0.6B

--origin-hf-dir gives the converter the configuration and the tokenizer of the base model. The Megatron trainer writes its own format, and section 6.1 names its converters.

Turn on fault tolerance

--use-fault-tolerance --ft-components rollout

--ft-components picks what Miles watches and replaces: rollout, train, or both. Without the flag, Miles watches rollout. Each part has limits in the Miles version we ran:

• Rollout fault tolerance does not work with a dedicated eval fleet, or with --fully-async --colocate.
• Train fault tolerance turns on independent data-parallel groups, and those need the Megatron trainer.

Miles main of 2026-10-20 changes this. It adds weight_update as a third part. It also allows rollout fault tolerance only with p2p weight transfer on a split layout. Read the flag help in your image before you copy a command.

What the Miles team reports

The Miles team built a fault injector for its tests. Their release notes report a run where a rollout engine crashed. Its loss curve matched a run without a crash, bit for bit. That is their test on their hardware. Repeat a crash test on your own setup before you trust it.

Known problems

• A report in the Miles issue tracker covers six agent example launchers. Each starts with a cleanup that stops every process with sglang in its command line. With MILES_SCRIPT_EXTERNAL_RAY=1, that cleanup also stops the Ray cluster that you started. With those launchers, remove the cleanup or give Miles its own Ray cluster.
• After an engine dies without fault tolerance, the trainer keeps sending requests to it and never fails. Add a watchdog to every run.

Miles runs inside a container image that holds Miles, a patched SGLang, the trainer libraries, and prebuilt GPU kernels. The kernels are the part that ties an image to a GPU.

When you finish this section, you can choose a Miles image for your GPU and record exactly which one a run used.

Start from the official image

The Miles team publishes radixark/miles with a tag per release and a latest tag. Each image is built for CUDA 13, for both linux/amd64 and linux/arm64.

At release v0.1.2, the image builds on SGLang v0.5.21 with the Miles patches on top. So the SGLang inside the image is not a stock SGLang release. Read its version from the image, not from the SGLang project.

Check the kernels against your GPU

The image holds FlashAttention 2 and FlashAttention 3. The FlashAttention 3 build targets Hopper only. On any other GPU, a setting that needs FlashAttention 3 fails at start, as section 4.6 shows.

The Miles documentation lists NVIDIA GPUs from A100 to GB300, and AMD MI300X to MI355X. Our GB10 is not on that list. The arm64 image still ran on our GB10, with the FSDP trainer and the default FlashInfer attention.

Record what each run used

latest moves. Two runs a week apart can use different code under the same tag. Record three facts at the start of every run: the image digest, the Miles commit, and the SGLang version. Section 1.1 gives the three commands and our output.

Our latest image held Miles from 2026-10-15 and a development build of SGLang 0.5.22, newer than the v0.1.2 release.

To repeat a run, pull the image by its digest, not by its tag:

docker pull radixark/miles@sha256:c64fa344e3a86715f03c0a1b6a422c474e0035355591a2f6da87bf52a74c52da

The first six entries are problems we met on a GB10: five symptoms of failed runs and one timing effect of heat. The seventh uses our healthy runs as the reference. The eighth comes from the Miles and SGLang source and documentation. The others come from reports in the Miles issue tracker. We have not reproduced the eighth or the others.

When you finish this section, you can find the cause of a failed or wrong run from its first symptom.

The GPU is idle and the log keeps growing

• Cause: the rollout engine died, and the trainer keeps sending requests to it.
• Check: count 503 Service Unavailable in each of the last few minutes of the log. Our failed run wrote more than 2,600 in every full minute for an hour. A healthy run with 16 answers for each prompt wrote short bursts of up to a minute, then none.
• Fix: stop the run, then read the engine lines above the first 503 error. Start every unattended run with the watchdog in section 1.2.

The run hangs after a reward import error

• Cause: Miles cannot import the module that --custom-rm-path names, usually because PYTHONPATH does not reach the lab folder.
• Check: look for ModuleNotFoundError with the name of your reward module in the log. No rollout 0 line follows it.
• Fix: the run does not stop by itself, and the watchdog does not stop it, because the engine writes only a few 503 errors. In our test the job stayed up for two hours. Stop the container, fix PYTHONPATH as in section 3.5, and start the run again.

The engine stops at start with a FlashAttention 3 error

• Cause: the run asked for FlashAttention 3, usually through true on-policy mode, on a GPU outside its range.
• Check: look for FlashAttention v3 Backend requires SM>=80 and SM<=90 in the engine log.
• Fix: on a non-Hopper GPU, drop true on-policy mode and use the default backend. Section 4.6 explains the limit.

The job stops with a Ray OutOfMemoryError

• Cause: on unified memory, Ray reads one memory for the whole node, and the node is close to full. Ray stops a worker above its threshold.
• Check: the line Memory on the node ... was 115.86GB / 121.69GB or similar, and no CUDA out-of-memory error.
• Fix: use a smaller model or less work per step, and leave the Ray threshold at its default. A lower engine memory fraction did not help our 4B run. Section 2.3 has the details.

The trainer rejects the attention implementation

• Cause: a trainer attention name that the installed transformers library does not accept, such as triton.
• Check: a ValueError that lists the attention implementations transformers accepts.
• Fix: pick one from that list. On our GB10 we left the trainer setting at its default.

Step times change from run to run

• Cause: thermal pauses or a lower clock on a hot GPU. We expect the samples and the rewards to stay the same, but we have not tested that.
• Check: the pause log of the heat guard from section 1.2, and the GPU clock during the run.
• Fix: time only steps without a pause, and compare times only within one session. Section 5.3 gives the rule.

The gap at step 0 is far above your range

• Cause: the engine and the trainer do not start from the same weights, or they read the same weights in different formats.
• Check: compare train/train_rollout_logprob_abs_diff at step 0 with your healthy runs. Ours were 0.0148 to 0.0157 for Qwen3-0.6B.
• Fix: stop the run before it trains on wrong numbers. Check the checkpoint paths of both sides.

The engine runs out of memory at start

• Cause: --sglang-mem-fraction-static does not fit the GPU. With --colocate, the trainer can already hold GPU memory when the engine starts. A fraction that is too high leaves no room for activations and CUDA graphs. A fraction that is too low leaves no room for the KV cache.
• Check: a CUDA out-of-memory error in the engine lines during startup or on the first batch, and no Ray OutOfMemoryError. A fraction that is too low does not always stop the engine. Read #tokens in the KV Cache is allocated line. In our check at 0.02, the engine started with 3,082 tokens, and the first rollout took 386 s against 61 s at 0.25.
• Fix: lower the fraction for the first case, or raise it for the second, by 0.05 or 0.1 at a time. The Miles documentation asks for 0.8 or lower with --colocate. Our runs used 0.4. SGLang 101, section 3.2 explains how the engine divides its memory.

A reward function with fractions moves a group that should stay still

• Cause: a report in the tracker shows that a group of 16 answers, each scored 0.3, got small nonzero advantages from float32 rounding. Binary rewards of 0 and 1 do not hit this.
• Check: a group where every answer has the same reward but the advantages are not 0.
• Fix: use rewards that a float32 can hold exactly, or check the advantages of such groups yourself.

Training goes wrong after offload on FSDP

• Cause: a report in the tracker shows that offloading the trainer state with FSDP2 on two or more GPUs can corrupt shards of uneven size.
• Check: nonfinite values, a jump in the loss or the gradient norm, or a collective timeout after the first offload. The model has layers that do not split evenly across ranks.
• Fix: run once with --no-offload-train and compare. Follow the issue for the code fix.

The weight update fails on many GPUs

• Cause: two reports in the tracker. A colocated update with pipeline stages crashed in NCCL 2.29 and 2.30 with a tuner plugin, and NCCL 2.31 fixed it. Deterministic serving on two GPUs set a different number of NCCL channels than the trainer, and the transfer failed with a size mismatch.
• Check: a segmentation fault or a Message truncated error during the weight update.
• Fix: update NCCL to 2.31 or later for the first case. For the second case, turn off deterministic serving on multi-GPU engines until a fix is released.

Rollout gets slow with hundreds of open requests

• Cause: a report in the tracker measured a client cost that grows with the number of open requests. It was 4.8 s at 256 open requests and 43.7 s at 1,536.
• Check: rollout time that grows faster than the number of samples.
• Fix: keep the number of open requests per engine lower, or follow the issue for the client change.

The launcher stops a Ray cluster you started yourself

• Cause: a report in the tracker shows that six agent example launchers stop every process with sglang in its command line. With MILES_SCRIPT_EXTERNAL_RAY=1, that stops your Ray cluster too.
• Check: your other Ray jobs stop when a Miles run ends.
• Fix: remove the cleanup from those launchers, or give Miles its own Ray cluster until the issue is fixed.

Two weeks into a project, you will want to know why last Tuesday's run learned faster than today's. You can only answer that if both runs recorded the same facts.

When you finish this section, you can record each run so that you can compare it later. You can also replay a step to debug it.

Record these facts for every run

Keep the engine log next to the trainer log. Of the four failures we met, one began in the engine and three began in the trainer, so you need both.

Change one thing per run

Compare two runs only when one setting differs. Our first offload run changed the offload setting and the engine memory fraction together, and it showed a 12 percent faster rollout. A same-day pair that changed only the offload setting showed 1 to 3 percent. Section 2.2 has both.

Compare times on the same step numbers. The rollout time follows the answer length, and the answers get shorter as the model learns. A median over different steps can differ for that reason alone, as section 5.3 shows.

Save what you need to replay a step

Miles can save the data of a run so that you can replay one part without the other:

• --save-debug-event-data writes the events of each step. Our runs kept these for every run.
• --save-debug-rollout-data writes the rollout samples. Load them with --load-debug-rollout-data and --debug-train-only to train on the same samples again, with no engine.
• --debug-rollout-only runs the engine side alone.

Replay is the fastest way to test a loss or a reward change. Every replay sees the same samples, so a difference comes from your change and not from the sampler.

Reinforcement learning on a language model spends most of its time generating text. A training library can generate, but slowly: one sequence at a time, with no shared cache. An inference engine generates fast, but it was built to serve users, not to agree with a trainer.

When you finish this section, you can explain why Miles runs its own inference engine, and name the five ways the two sides can disagree.

Miles puts an SGLang engine under the trainer's control. This page explains why, and what that control has to cover. The Inside Miles guide walks one step from start to end, so this page stays with the reasons.

What RL asks of the engine

A GRPO step generates many answers for each prompt. Our runs generated 8 answers for each of 32 prompts, 256 answers per step. Three things make that fast:

• Batching. The engine runs hundreds of sequences in one pass.
• A shared prompt cache. The 8 answers of one prompt share the same prompt, so the engine can compute it once and reuse it.
• Short answers end early. A finished answer leaves the batch, and its memory goes to the next one.

A training library is not built for these. So an RL framework that cares about speed needs a real engine.

What the engine must do for RL

A serving engine never changes its weights while it runs. An RL engine must:

• Take new weights after every training step, in seconds.
• Give its memory back while the trainer runs on the same GPU, and take it back after.
• Pause or stop the requests that are still running when the weights change.
• Return the log probability of every token it sampled.

Each of these needs a hook into the engine. That is why Miles runs a patched SGLang build, and why the SGLang inside the image is not a stock release.

Five ways the two sides disagree

GRPO assumes that the trainer and the engine agree on the probability of each sampled token. They can disagree in five ways:

Most of our runs on one GB10 had only the numbers source. One run sampled with top-p and top-k, so it also had the sampling source. In our Qwen3-0.6B runs, the gap stayed at 0.013 to 0.016 per token. Section 8.2 matches each correction to its source.

Our view: most of what makes an RL framework hard is not the loss function. It is keeping these five sources small and measured while the engine runs at full speed.

Section 8.1 names five sources of disagreement: tokens, routing, sampling, numbers, and staleness. Miles has a method for each. Some methods remove a cause, and others only reduce its effect on the gradient.

When you finish this section, you can explain which correction fixes which cause, and which of them we measured.

Two kinds of method

A method that removes a cause makes the two sides compute the same thing. After it, that part of the gap is gone.

A method that reweights leaves the gap in place and changes how much each token counts in the loss. It keeps a token that the two sides disagree on from pulling the update too far.

Remove causes first, because every reweighting adds variance to the gradient.

The methods

Truncated importance sampling: each token gets the ratio of the trainer probability to the engine probability as a weight, and the weight is capped. The cap keeps one token from dominating the update.

Score centering: a separate loss that uses the top-k probabilities of the engine's sampler to correct the score of off-policy samples. It turns on an FP32 output layer in the engine, which costs serving compute and memory.

What we measured

On our GB10 with Qwen3-0.6B, two of these touched our runs:

• Sampling replay lowered the gap by 11.5 percent, from 0.0153 to 0.0135. Section 4.4 has the run.
• Deterministic inference is not on the list, because it removes no cause of the gap. It made the engine repeat itself, and the gap did not move. Section 4.5 has the runs.

We have not run the other methods. Most of them only matter with a MoE model, multi-turn rollouts, low precision, async training, or a Hopper GPU.

Our view

Read the gap before you pick a method. On a dense model with plain sampling, a flat gap of about 0.015 is the kernel difference of that setup. A reweighting method only hides it. A gap that grows after a change points to a cause you added: a new sampler, a MoE model, FP8 rollout, or older samples. Match the method to that cause.

GRPO assumes that the probability the engine gave a token is the probability the trainer computes for it. In every healthy Qwen3-0.6B run on our GB10, the two sides disagreed by 0.013 to 0.016 per token on average. This page explains where that gap comes from, and why closing it takes a contract and not a flag.

When you finish this section, you can explain why true on-policy mode covers only some models and GPUs.

Two programs compute the same model

The engine and the trainer hold the same weights after each update. They still run different programs. The engine is built for speed: it batches many requests, picks kernels by batch size, and fuses operations. The trainer is built for gradients: it runs a different attention kernel and a different order of operations.

Floating-point addition gives slightly different results in a different order. The same long sum, added in two orders, can differ in its last bits. Across 28 layers and a vocabulary of 151,936 tokens, those bits add up. We think that this is the gap we measured.

So the gap is not a bug in either side. It is the cost of two programs that each do their own job well.

Deterministic inference fixes the wrong half

Deterministic inference makes the engine give the same result for a request, whatever the batch around it. That makes the engine repeat itself. It does not make the engine compute like the trainer.

Our runs show this directly. With deterministic inference on, two runs gave the same gap at step 0, to six decimal places, and the gap stayed at 0.015. Section 4.5 has the numbers.

What the contract pins

To get a gap of exactly 0, both sides must run the same arithmetic in the same order. The contract in Miles pins that down for one family of models:

• The same attention kernel on both sides, FlashAttention 3.
• Operations that give the same result at any batch size.
• Log probabilities computed the way the engine computes its prefill.
• A profile per model, because each model has its own layers to match.

Each item removes one source of different ordering. Together they make the two programs agree bit for bit.

Why it stops at Hopper and at Qwen3

Every pinned item is a piece of code that someone checked on specific hardware. FlashAttention 3 is built for Hopper GPUs. The batch-invariant operations and the model profile were checked for dense Qwen3 models. A new GPU or a new model needs its own kernels, checked the same way.

That is why we call it a contract and not a setting. A setting asks the code to try. A contract lists the cases where the code has been proven, and refuses the rest.

Our gap is the floor of our setup

Our gap of about 0.015 is the floor of one setup. That setup is BF16 on both sides, SGLang kernels in the engine, and FSDP kernels in the trainer. It is not a law of RL. Other setups move it. We did not measure the changes below. They come from published work:

• Precision. A 2025 study ran the trainer and the engine in FP16 and reported that the gap almost went away. FP16 keeps more bits of each number than BF16, so each rounding is smaller.
• Tensor parallel size. A 2025 paper showed that an engine split over several GPUs adds in a different order than a trainer on one GPU. That gap stays even with batch-invariant kernels, so the tensor parallel size of the engine is a source of its own.
• Bitwise kernels. The same paper then made a vLLM engine and an FSDP trainer agree bit for bit, with kernels that add in one fixed tree order. In their test on Qwen3-8B, inference took 22 to 63 percent longer.

Other teams publish their healthy gaps too. Use them as a reference for the size of a working gap, not as a target for your run:

Our view: on a GPU outside the contract, first decide if the gap matters for your run. If it does, precision, tensor parallel size, and bitwise kernels are the levers, and each one costs speed or work. If it does not, measure the gap at step 0 and keep it inside the range of your healthy runs. Then remove the part that the sampler adds.

On one GPU, the trainer and the rollout engine cannot both hold all their memory at once. Miles makes them take turns. This page explains what each turn gives back, what the weight update does, and why a GB10 changes both stories.

When you finish this section, you can explain what offload and the weight update do on a machine with shared memory.

Two sides take turns

With --colocate, the trainer and the engine share the same GPU. Each step has the same order:

1. The engine generates the answers for the step.
2. The engine releases its memory.
3. The trainer loads its state, computes the log probabilities, and trains.
4. The trainer releases its state.
5. The engine loads again and receives the new weights.

The --colocate flag also turns on offload for both sides. --offload-rollout-level decides what the engine gives back. The default is both of its memory tags: kv_cache, the cache of attention keys and values, and weight, the model weights.

Offload on a normal machine and on a GB10

On a machine with a separate GPU, offload copies bytes from GPU memory to CPU memory over the bus. The copy is the cost.

On a GB10 there is no second memory. GPU memory and CPU memory are the same 128 GB. So offload frees the GPU allocation, and the bytes stay in the same pool. For Qwen3-0.6B, the trainer moved its state to host memory in a median of 0.76 s and back in 0.87 s, over 48 steps. That is small next to a step of more than two minutes.

The engine release before training has no timer. The engine resume after the weight update is in perf/finalize_and_resume_engines_time.

Our view: on a GB10, part of the trainer offload and onload time is probably setup work, not copying. We have not measured that split.

The weight update stays inside the GPU

After training, the engine needs the new weights. Under --colocate, the trainer shares handles to its GPU tensors with the engine, and the engine copies from them. The bytes never travel over a network.

The trainer sends the weights in buckets. --update-weight-buffer-size sets the bucket size, and our runs used 512 MiB. The update took 0.64 s for Qwen3-0.6B and 0.91 s for Qwen3-1.7B, under one percent of a step. Section 2.4 has the table.

One pool changes what Ray sees

Shared memory also changes what other tools see. Ray reads the memory of the whole node. On a GB10 that number includes the model weights, the optimizer state, and the KV cache. So Ray can stop the trainer before any single GPU allocation fails. On a GB10 that stop is a real warning, because the node is close to full for every process. Section 2.3 shows the kill and how to lower the memory the run needs.

What we do not know yet

Keeping the engine loaded between phases saved little rollout time on our GB10. We ran a same-day pair on one seed. On the steps that were clean in both runs, it saved 0.6 to 1.8 s of a rollout of about a minute. Section 2.2 has the runs.

Every RL recipe changes something: how answers are generated, how they are scored, which groups the trainer keeps. If each change needed a fork, every team would carry its own copy of Miles. Miles avoids that with a set of flags that each take a Python import path.

When you finish this section, you can explain where a custom function plugs into a step, and choose the smallest hook for your change.

How a path becomes a function

Each of these flags takes a path like my_package.my_module.my_function. Miles imports the module and takes the function from it. Miles also checks a small registry first, but only its own tests fill that registry.

So your function lives in your own file, outside the Miles source. Mount the file into the container, or install it as a package, and pass its path.

The hooks, in the order a step uses them

Use the smallest hook

Each hook down the table replaces less of Miles. Pick the lowest one that covers your change:

• A new scoring rule needs only the reward hook.
• A multi-turn or tool-calling agent needs the generate hook. Section 3.3 shows the built-in ones.
• A DAPO-style filter needs only the group filter. Miles has one built in that drops groups where every answer got the same reward.
• Replace the whole rollout only when the step itself changes shape.

A smaller hook keeps the rest of Miles working for you: batching, the metrics, and the gap checks. Partial rollout keeps working only when the generate function supports it. The built-in multi-turn and agent generate functions do not, as section 4.2 shows.

Fork only when no hook fits

Some teams fork Miles or slime to add a hook that does not exist yet. The Miles issue tracker shows that pattern: a contributor fork with a regression test, then an issue. If you fork, keep the change small and send it back, so that your fork does not drift from the main line.

Every rollout request goes through a router before it reaches an engine. On one GPU with one engine, the router has one place to send each request. With many engines, or with agent sessions, where it sends a request decides how well the engines reuse their caches.

When you finish this section, you can explain what the router does in a Miles run, and when to change it.

The two routers

Miles starts one router for each model in the run, with no GPU of its own. Since release v0.1.1, Miles always starts its own router, and you cannot point a run at a router you started yourself.

The default SGLang router comes from a build that the Miles team keeps for Miles. That build passes RL fields through the router to the engine and back. Two examples are the sampled token set for sampling replay and the top log probabilities.

Our runs used the default router. The launch line in our logs shows it:

sglang_router.launch_router --log-level warn --prometheus-port 20036 --request-timeout-secs 14400 --host 172.17.0.3 --port 20034

Routing for agent sessions

With --use-session-server, the router keeps every turn of one conversation on the same engine. Each turn then reuses the cache of the turns before it. Miles sets the routing policy for this case by itself.

With attention data parallelism on the engines, Miles makes the router aware of the data-parallel ranks. The router can then pick a rank inside an engine.

When to change the router

• On one engine, keep the default. There is nothing to balance.
• With many engines and single-turn prompts, keep the default.
• With agent sessions, turn on the session server, and let Miles set the policy.
• Use the Miles router only when you need to send text requests. It cannot route split prefill and decode.

When requests fail, read the router log next to the engine log. A request that fails at the router never reaches an engine.

Miles makes more sense when you know its parent. It began as a copy of slime, an RL framework from the GLM team at Zhipu. It still shares most of its design with slime.

When you finish this section, you can explain how Miles relates to slime, and what that means when you upgrade.

slime first

slime started in June 2025. It joined a Megatron trainer to SGLang rollout engines, with a data buffer between them for custom data generation. The GLM team uses it to train its GLM models with reinforcement learning, from GLM-4.5 onward.

The split

RadixArk is a company that builds on SGLang and says it will keep investing in it. Its first post said Miles would focus on new hardware, large mixture-of-experts RL, and stability in production. It also said that most additions would go back to slime.

How far they have grown apart

By 2026-10-20, Miles had about 2,500 commits that slime does not have, and slime about 1,200 that Miles does not have. The features that Miles has and slime does not, at that date:

• A third trainer backend, torchtitan, next to Megatron and FSDP.
• Weight transfer over RDMA, and an update driver shared by every transfer method.
• A session server that passes tokens in and out for agent turns.
• MXFP8, NVFP4, and INT4 training recipes.
• Multi-LoRA training.
• An experimental Kubernetes backend, and a fault injector for its tests.

slime went its own way too. Its v0.3.0 added an agent module. Its v0.4.0 kept only the synchronous and the fully async modes, and added its own storage for rollout data.

What this means for you

• Read the Miles release notes, not the slime ones. A flag that exists in slime can differ in Miles, and the other way around.
• Use the SGLang inside the image. Miles runs a patched SGLang with hooks for RL. A stock SGLang release does not have all of them.
• Pin the image. Miles merged hundreds of pull requests in each release cycle. The latest tag moves, so record the digest of every run, as section 6.4 shows.

The value column gives the setting of our lab runs where we set it. A dash means our runs did not set the flag.

Model and data

Rollout

Reward and loss

Placement and memory

Engine

Checks and debugging

The ranges come from our lab runs: Qwen3-0.6B on GSM8K on one GB10, 32 prompts with 8 answers each, a limit of 1,024 tokens. They hold for that setup. Build your own ranges from two or three seeds of yours.

Rollout

Training

Time

Time values hold only for steps that no heat-guard pause touched. The lab notes give the rule.

Evaluation and async

We did not run evaluation or async training, so this table has no ranges.

This page is reference. It lists how each number in the manual was made, so you can repeat the run or judge the result.

Machine and image

Default run settings

Every run used these settings unless its name says otherwise:

Runs

Four runs failed and are left out of every table: two true on-policy runs that stopped at start, and two Qwen3-4B runs that Ray stopped. Our run queue gave each run a time limit, and stopped the container when the limit came. The e2-n16 and e2b-dapo-filter runs had finished all their steps and were shutting down at that time. The summary marks them as partial.

The heat guard

The GB10 runs hot under sustained load, so a script pauses the Miles container when the GPU reaches 84 C. It resumes the container when the GPU is at 74 C or below after at least 45 seconds, or after 120 seconds at most. On its first day the limits were 80 C and 68 C, and they changed to 84 C and 74 C that morning. When the guard stops, it resumes a paused container first, so no run stays paused.

The pause log holds 1,005 pauses, each with its resume, up to 2026-10-22.

The timing rule

A pause adds wall-clock time to the phase it falls in. So every time in this manual comes from steps that no pause touched, with one exception. The DAPO filter run of section 3.2 had no clean step, so its rollout times take the pause time out of each step instead:

• Rollout times: the rollout window ends at the timestamp of the rollout metrics line and lasts perf/rollout_time. A step counts when that window overlaps no pause.
• Whole-step times: the window ends at the timestamp of the trainer metrics line and lasts perf/step_time. A step counts when that window overlaps no pause.

The summary file stores both medians, all steps and clean steps, with the number of clean steps next to each. Under the heat guard, few whole steps escape a pause. So whole-step times in this manual come from plain and plain-48. Both ran before we added the heat guard, so no pause could touch them. plain-48 ended at 00:11 on 2026-10-18, and the guard logged its first pause at 10:57 that day.

Check the stored outputs

The capture kit checks the stored outputs without a GPU. It replays the watchdog rule on the stored error times of the 16-answer run. It also marks the pauses again from the pause log, and it computes every median again from the stored steps. The raw run logs stay on the GB10. With those logs, the kit can also build the summary again and compare it with the stored one.

Compare the same steps

A median over clean steps can still mislead. The default setting gave a 47.5 s rollout in plain-48 and 59.1 s in the 6 clean steps of e6a-base. On the same 6 step numbers, plain-48 took 60.9 s. The rollout time follows the answer length, so compare times on the same step numbers.

Results from one run

These numbers rest on one run, and each page that uses them says so:

• The rollout time with the engine kept loaded, in section 2.2. A same-day pair gave 1 to 3 percent on one seed.
• The cost of deterministic inference, in section 4.5.
• The group sizes of 4 and 16 answers, in section 3.1.
• The reward and the rollout cost of the DAPO filter, in section 3.2. Its rollout times take the pause time out, because no step was clean.

Why the first offload pair does not hold

Our first pair compared e6a-base with e3c. It showed a clean rollout median of 59.1 s for the default and 52.0 s with the engine kept loaded, a gain of 12 percent. The two runs also used different engine memory fractions, 0.4 and 0.25, and their clean steps were different steps.

Step 10 was clean in both runs. The engine kept loaded took 46.7 s with answers of 599 tokens. The default took 56.3 s with answers of 695 tokens. The rollout time follows the answer length, so the shorter answers explain most of that gap. The same-day pair e3e and e3f then changed only the offload setting, and gave 1 to 3 percent.

When two sources disagree, the one higher on this list wins, and the page says so.

1. Our runs

Every number marked as measured comes from runs that we recorded on one NVIDIA DGX Spark with a GB10 GPU, from 2026-10-17 to 2026-10-22. The lab notes list the runs, the image, and the timing rule.

2. The Miles code we ran

Every flag and default that a page names comes from the Miles code inside the image we ran: Miles main from 2026-10-15. We checked the parts that pages depend on against Miles main from 2026-10-20.

3. Miles documentation and release notes

Documented behavior comes from the Miles documentation and the release notes of v0.1.0, v0.1.1, and v0.1.2. Where the documentation and the code disagree, we follow the code. One example: the help text of the custom reward flag shows a plain function, and the code needs an async one.

4. The Miles issue tracker

The troubleshooting entries that we did not reproduce come from reports in the Miles issue tracker. Each one is marked as a report, not as our result.

5. Papers and posts

• The Miles v0.1 technical report, arXiv 2609.08368, September 2026.
• Rollout Routing Replay, arXiv 2510.11370.
• Score Centering Stabilizes Off-policy Reinforcement Learning, arXiv 2609.20807.
• The first public Miles post, November 2025, and the Miles v0.1 post, August 2026, both from the RadixArk team.
• The slime repository and its README, for the history of the project.
• Defeating the Training-Inference Mismatch via FP16, arXiv 2510.26788, October 2025.
• Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatch, arXiv 2511.17826, November 2025.
• Jet-RL, on-policy FP8 reinforcement learning, arXiv 2601.14243, January 2026.
• Scaling Laws for Collapse in Asynchronous GRPO, arXiv 2607.01083, July 2026.
• Unified FP8 for MoE RL, an LMSYS post by the Miles team and its partners, November 2025.
• The vime announcement, a vLLM team post, June 2026.
• SGLang pull request 42408, DeepSeek-V4.1 on GB300 with Miles, October 2026.
• The SGLang speculative decoding documentation.
• The SkyRL quantized rollouts documentation, 2026.
• TRL issue 6789, the importance-sampling bias of truncated sampling, August 2026.

Results that other teams measured appear on the pages with the team named, and with the hardware they used.

Each row links to its figure, and the claim is the claim: field of each figure block. Every measured number in a figure comes from our runs on one DGX Spark. The charts read the summary of those runs when the manual is built. The colours follow the palette of the front matter:

• blue is the trainer: the FSDP actor, its weights, and its optimizer
• violet is the rollout engine: the SGLang server that writes the answers
• green is data and rewards: prompts, answers, groups, and scores
• amber is a lifecycle step, such as a phase of a training step or a weight update
• teal is durable state: checkpoints, logs, and the summary files
• rose is a failure: a dead engine, an out-of-memory kill, or a crash
• grey is outside Miles: the host, Docker, and Ray