Split rollout and training across GPUs

On one GPU, the trainer and the engine take turns, and the weight update is a copy inside the GPU. On many GPUs you have a choice. Keep both sides on the same GPUs and take turns, or give each side its own GPUs and send the weights across.

When you finish this section, you can choose a placement and a weight transfer method for a run on many GPUs.

We cannot measure this on our single GB10. This page comes from the Miles source and documentation.

Choose a placement

Set the sizes with these flags:

• --actor-num-nodes and --actor-num-gpus-per-node set the trainer GPUs.
• --rollout-num-gpus sets the rollout GPUs in a split layout.
• --rollout-num-gpus-per-engine sets how many GPUs each rollout engine uses.

Start colocated. Split only when the rollout phase takes most of the step and you have the GPUs to give it. A split layout leaves trainer GPUs idle during rollout unless you also run async training, as in section 4.1.

Choose how the weights move

In a split layout, the weights leave the trainer GPUs after every update. --update-weight-transfer-mode picks the method:

Under --colocate, Miles uses a copy inside the GPU instead, and the transfer mode does not apply.

Choose the trainer

This manual uses the FSDP trainer, because it loads a Hugging Face model folder as it is. Miles has two other trainers, and --train-backend picks one:

Some features in this manual work only on Megatron: fully async training with --colocate, train fault tolerance, MTP training, and the step-0 checks of --ci-test. The Miles documentation recommends Megatron for large models and for jobs on many nodes.

torch_dist: the checkpoint format of Megatron. It does not depend on the parallel layout, so you can change the layout later without a new conversion.

Convert a model once before the first Megatron run. The architecture flags come from the model file of Miles for that model, here qwen3-4B:

cd /root/miles
MODEL_ARGS=$(python3 miles/utils/external_utils/model_args_utils.py qwen3-4B)
PYTHONPATH=/root/Megatron-LM python3 tools/convert_hf_to_torch_dist.py $MODEL_ARGS --hf-checkpoint /root/lab/models/Qwen3-4B --save /root/lab/models/Qwen3-4B_torch_dist

Run these lines inside the container, with the lab folder mounted as in section 1.1. Our launcher builds an FSDP run only. For Megatron, start from a launcher of Miles for your model, in the scripts folder of the Miles source. We have not run the Megatron or the torchtitan trainer on our GB10. We ran only the conversion to torch_dist for Qwen3-0.6B, and it wrote the checkpoint folder.

Start a run on more than one node

Every node must be in one Ray cluster before Miles submits the job. The launchers of Miles do it in one of two ways:

• One command for each node role. A worker command joins the cluster of the head node and waits. A train command starts the head node, waits for all GPUs, and submits the job.
• One command on the head node, which connects to the other hosts over ssh and adds them to the cluster.

To use a Ray cluster that you started yourself, set MILES_SCRIPT_EXTERNAL_RAY=1. The launcher then skips its own ray start and only submits the job. Set MASTER_ADDR to the address of the head node on every node, so that Ray and the trainer use the right network interface. Then set the trainer size with --actor-num-nodes and --actor-num-gpus-per-node.

Section 6.3 lists a known problem of some launchers with a Ray cluster of your own. We have not run Miles on more than one node.

What to measure after the split

• perf/update_weights_time against perf/step_time. On many machines the transfer is no longer a copy inside one GPU.
• The idle time on each side, from perf/train_wait_time and the rollout timers.
• The gap between the trainer and the engine, as in section 5.2, after the first update with the new method.