Collective Ops From Scratch
The four collective operations that hold distributed training together are allreduce, broadcast, allgather, and reducescatter. Every other primitive a training framework offers is a wrapper around these. Build them once over a multiprocessing.Queue mesh, verify them against a reference implementation, and the rest of the track becomes plumbing. Implement ring allreduce in two passes (reduce-scatter then allgather) and prove the per-rank communication volume is 2(N-1)/N bytes per element. Build broadcast, allgather, and reducescatter on top of point-to-point sends over multiprocessing.Queue. Verify every primitive against a torch.distributed gloo reference for the same input. Defend the choice of ring versus tree on cluster shape, latency floor, and bandwidth ceiling. A naive allreduce over N ranks sends N times the tensor to a root and broadcasts N times back. Bandwidth scales as O(N) per rank, the root becomes a bottleneck, and the wall-clock floor is the slowest link times N. Ring allreduce flattens that into 2(N-1) chunks of size T/N, so per-rank bytes drop to 2T(N-1)/N independent of cluster size. Tree allreduce wins on small N and high-latency links because depth is log2(N) hops instead of 2(N-1). Pick the wrong topology for the cluster shape and the slowest GPU dictates step time. Every distributed training framework you will read this track depends on these four primitives. PyTorch DDP synchronises gradients with one allreduce…
Collective Ops From Scratch: The four collective operations that hold distributed training together are allreduce, broadcast, allgather, and reducescatter.…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.