Pipeline Parallel and Bubble Analysis
Tensor parallelism splits the matrix multiply across ranks. Pipeline parallelism splits the model across ranks, one stage per rank. Microbatches flow through the pipeline. The empty time at the start and end is the bubble; minimising it is the whole craft. Split a sequential model into N stages and simulate a forward pipeline across N ranks. Schedule M microbatches through the pipeline using the GPipe schedule (forward-only fill, then backward) and compute the bubble fraction. Compare bubble against the interleaved 1F1B schedule used in Megatron-LM and PipeDream. Defend stage assignment: equal compute per stage matters more than equal parameter count per stage. A 70B-parameter model in fp16 needs 140 GB of parameters alone. No consumer GPU holds it. ZeRO-3 shards parameters across ranks but still needs every rank to allgather the full layer for each forward step, paying log(N) hops per layer. Pipeline parallel takes a different route: cut the model into N stages and put one stage on each rank. Forward of layer 1 finishes on rank 0 and hands the activation tensor to rank 1; rank 1 runs layer 2 and hands to rank 2; and so on. Backward flows in reverse. Memory drops linearly because each rank only holds one stage; compute is sequential, which is the bubble problem. The bubble is the idle time at the…
Pipeline Parallel and Bubble Analysis: Tensor parallelism splits the matrix multiply across ranks. Pipeline parallelism splits the model across ranks, one…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.