DDP, FSDP, ZeRO and Pipeline Parallel

Pick a parallelism, size the memory, wire the collectives and fix the hangs. PyTorch gloo on CPU.

Four strategies side by side#

Factor	DDP	FSDP or ZeRO-3	Pipeline	Tensor parallel
Splits	the batch	params, grads, optimizer	the model by layers	each weight matrix
Per-GPU memory	full copy, no drop	1/N of each state	one group of layers	a slice of each layer
Traffic	all-reduce of gradients	all-gather per layer, reduce-scatter	activations at layer boundaries	all-reduce after each matmul
Link	overlaps with compute	more than plain DP	low bandwidth, across nodes	NVLink inside a node
Main cost	memory	gather every forward pass	pipeline bubbles	latency after every layer

Factor	DDP	ZeRO-1	ZeRO-2	ZeRO-3
Sharded	nothing	optimiser state	optim + grads	optim + grads + params
Memory per rank	params + grads + optim	params + grads + optim/N	params + grads/N + optim/N	params/N + grads/N + optim/N
Comm per step	1x allreduce	1x reduce_scatter + 1x allgather	1x reduce_scatter + 1x allgather	allgather and reduce_scatter per layer
Savings	none	~4x reduction	~8x reduction	~Nx reduction
PyTorch FSDP name	not listed	not listed	`ShardingStrategy.SHARD_GRAD_OP`	`FULL_SHARD`

Factor	Ring allreduce	Tree allreduce	Allgather	Reduce_scatter
Per-rank bytes	2T(N-1)/N	T log2(N)	T(N-1)/N	T(N-1)/N
Steps	2(N-1)	2 log2(N)	N-1	N-1
Use when	large T	small T, high latency	ZeRO unshard	ZeRO gradient sharding
Message size	above ~1 MB	below ~1 MB	any	any

dist.init_process_group(

backend="gloo",

init_method=f"file://{init_file}",file rendezvous

rank=rank,

world_size=world_size,

)

torch.manual_seed(SEED)same init on every rank

ddp = DistributedDataParallel(model, world_size=world_size)broadcasts params

for step in range(steps):

offset = step * world_size * batch + rank * batchown slice

optim.zero_grad(set_to_none=True)

loss.backward()

ddp.sync_grads()all_reduce SUM, then div_ by world_size

optim.step()

Symptom	Cause	Fix
Ranks diverge from step one	Each rank uses its own init	`dist.broadcast(tensor, src=0)` at construction
All ranks see the same batches	One shared seed for shuffling	`torch.manual_seed(seed + rank)` for shuffling
Allreduce deadlocks	Skipped params have no gradient	`find_unused_parameters=True`
Accumulation is slow	Allreduce on every microbatch	Wrap microbatches in `no_sync()`
LR breaks when N changes	SUM without the division	Divide by world_size
Pipeline hangs on the wire	Every stage sends before receive	Even ranks send first, odd ranks receive first
One stage gates every cycle	Stages split by parameter count	Equalise FLOPs per stage
70B FSDP over 80GB on 8 GPUs	105 GB per GPU	16 GPUs or activation checkpointing

Factor	1F1B	Zero Bubble	DualPipe	DualPipeV
Param copies	1	1	2	1
Bubble vs micro-batches	grows	grows	constant	small growth
Overlap	minimal	partial	full	partial
Simulator	`bubble_1f1b`	`bubble_zero_bubble`	`bubble_dualpipe`	`bubble_dualpipev`

Task	Code
Set the rendezvous port	`os.environ["MASTER_PORT"] = str(port)`
Start a CPU group	`dist.init_process_group(backend="gloo", rank=rank, world_size=world_size)`
Same params on all ranks	`dist.broadcast(p.data, src=0)`
Sum gradients	`dist.all_reduce(p.grad.data, op=dist.ReduceOp.SUM)`
Mean gradients	`p.grad.data.div_(world_size)`
ZeRO-1 gradient shard	`dist.reduce_scatter(local_grad, chunks, op=dist.ReduceOp.SUM)`
ZeRO-1 params back	`dist.all_gather(gathered, self.master_shard)`
GPipe bubble	`bubble_fraction(num_stages, num_microbatches)`

Command	What it does
`python3 code/main.py`	Lesson 48: two gloo ranks, writes `outputs/ddp-demo.json`
`--world-size 4`	Lesson 48: four ranks. Param spread stays under 1e-3
`--no-write`	Lesson 48: skip the JSON file
`python3 code/main.py`	Lesson 76: queue mesh against gloo, byte counter
`python3 code/main.py`	Lesson 77: single process against DDP on 4 ranks
`python3 code/main.py`	Lesson 78: loss and the ZeRO-1 memory table
`python3 code/main.py`	Lesson 79: Gantt chart and bubble percentage