Gradient Clipping and Mixed Precision
The optimizer and schedule from the previous lesson assume gradients are sane. They usually are not. A single bad batch can spike the gradient norm by three orders of magnitude. Mixed-precision training amplifies this by introducing FP16 overflow on the loss side. This lesson builds the two safety belts that production training cannot ship without: gradient clipping to a configured global L2 norm, and a mixed-precision loop with autocast and GradScaler that detects NaN and Inf, skips the step cleanly, and logs the scaling factor for forensics. Compute the global L2 norm over all parameter gradients and clip in place when it exceeds a configured threshold. Wrap a training step in autocast plus a GradScaler so FP16 forward and backward passes survive overflow. Detect NaN and Inf in the loss or gradient, skip the optimizer step, and log the skip. Report the GradScaler's scaling factor every step so a long sequence of skips is visible immediately. A training run that ran clean yesterday produces a loss curve that goes vertical at step 8,217. The culprit is a single batch whose gradient norm is 4,200, twenty times the previous peak. Without clipping the optimizer applies a step that resets every learning the model had done in the previous hour. With a global L2 clip at norm 1.0, the same batch contributes…
Gradient Clipping and Mixed Precision: The optimizer and schedule from the previous lesson assume gradients are sane. They usually are not. A single bad batch…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.