Pick a model size for one GPU

When you finish this section, you can estimate the memory of a run before you start it. You can also choose the largest model that fits.

Our result: on one GB10 with 128 GB of shared memory, Qwen3-1.7B fit for 4 short steps with answers cut at 256 tokens. Qwen3-4B did not fit, even with those short answers. We did not train 1.7B with the 1,024-token recipe.

Count the trainer memory first

The trainer is the large part of a colocated run. Full fine-tuning with Adam keeps the model weights, the gradients, and two optimizer values for each weight. With mixed precision that is about 16 bytes for each parameter, before activations. The engine adds its own copy of the weights and its KV cache on top.

The estimate is a lower bound. Activations, the engine, and the process overhead come on top, and the measured node memory below shows how large that overhead is.

What happened with 4B

Ray stopped both of our 4B runs in the training phase of the first step, at 115.86 and then 119.18 GB of 121.69 GB. The second run lowered the engine memory fraction from 0.4 to 0.2, and the kill came anyway. Section 2.3 shows the kill message, why it happens on a GB10, and what to change.

Choose the size

1. Start with the smallest model that can answer your task.
2. Multiply the parameter count by 16 bytes. On our 128 GB GB10, the limit fell between 27 GB and 64 GB of this estimate. Use half of your GPU memory as a first guess.
3. Run 4 steps at your real answer length. Read the node memory with free -g on the host, or in the Ray kill line. On a GB10, nvidia-smi reports no memory size.
4. Move up one size only when the short run leaves room.

To train a model of 4B or more, you need more GPUs (Part 6) or a recipe that keeps less optimizer state.

Train a larger model with LoRA

LoRA: low-rank adaptation. The trainer keeps the model weights frozen and trains two small matrices next to some of the layers. Only those small matrices get gradients and optimizer state.

LoRA is the usual way to train a model whose full trainer state does not fit. In the Miles version we ran, only the Megatron trainer supports it. The FSDP trainer of this manual refuses LoRA at start. To try it:

1. Switch to the Megatron trainer, as section 6.1 shows.
2. Add --lora-rank 16, or another rank above 0. The default rank of 0 turns LoRA off.
3. Keep one adapter. Under --colocate, Miles trains one adapter, and several adapters need a split layout.

The engine serves the same adapter that the trainer updates. We have not run LoRA or the Megatron trainer on our GB10. So we do not know if a 4B model fits this way on 128 GB.