Gradient Accumulation
Train at an effective batch you cannot afford, one micro-batch at a time. Scale the loss, hold the optimizer step, and let the gradients pile up. Derive the effective batch identity: effectivebatch = microbatch accumsteps. Implement loss-per-micro-batch scaling so the accumulated gradient matches a single full-batch backward. Skip optimizer synchronization until the last micro-batch (sync-on-last-step). Read a throughput against effective batch curve and explain the diminishing return. You want to train at an effective batch of 512 because the loss curve is smoother and the optimizer step makes more sense at that scale. The accelerator on the desk holds 32 examples before it runs out of memory. Doubling the batch is not an option. Halving the model is not an option. The trick the field reached for in 2017 and never stopped using is to run 16 backward passes, let the gradients accumulate inside the parameter buffers, and only step the optimizer when the count reaches the target. The risk is that the loss is no longer the same number it was at the bigger batch. The cross entropy of 16 mini-batches summed naively is 16 times the loss of one full batch. Without scaling, the gradient direction is correct but the magnitude is wrong, and the optimizer step is 16 times too big. The fix is one division. The…
Gradient Accumulation: Train at an effective batch you cannot afford, one micro-batch at a time. Scale the loss, hold the optimizer step, and let the…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.