Sharded Checkpoint and Atomic Resume
A 70B-parameter training job is paused by a node failure every few hours. The checkpoint format decides whether you lose 30 minutes or 30 hours. A sharded checkpoint writes every rank's shard in parallel and records ownership in a manifest. Resume loads each rank's shard from its own file, reconstructs the state on the same world size, and the optimiser steps as if nothing happened. Atomic write keeps a half-finished checkpoint from poisoning the next resume. Save a multi-rank checkpoint as a per-rank shard file plus a manifest that records which rank owns what. Use the atomic write pattern (write to a temp path then rename) so a crash mid-write never produces a half-finished checkpoint. Resume from the manifest, verifying byte-equal state for both fp16 parameters and the ZeRO optimiser state on every rank. Defend the manifest schema against the three failure modes: world-size change, shard count mismatch, and partial write. A vanilla checkpoint reads all parameters and optimiser state into rank 0, gathers, and writes a single file. For a 70B model that is 1.1 TB of state through one rank's network port. The write blocks every other rank because they idle waiting for the gather. The IO bandwidth is the slowest single GPU's network link, not the aggregate. On a real cluster the gather-then-write step can take longer…
Sharded Checkpoint and Atomic Resume: A 70B-parameter training job is paused by a node failure every few hours. The checkpoint format decides whether you lose…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.