Learning Rate Schedules and Warmup
The learning rate is the single most important hyperparameter. Not the architecture. Not the dataset size. Not the activation function. The learning rate. If you tune nothing else, tune this. Implement constant, step decay, cosine annealing, warmup + cosine, and 1cycle learning rate schedules from scratch. Demonstrate the three failure modes of learning rate selection: divergence (too high), stalling (too low), and oscillation (no decay). Explain why warmup is necessary for Adam-based optimizers and how it stabilizes early training. Compare convergence speed across all five schedules on the same task and select the appropriate one for a given training budget. Set the learning rate to 0.1. Training diverges -- loss jumps to infinity in 3 steps. Set it to 0.0001. Training crawls -- after 100 epochs, the model has barely moved from random. Set it to 0.01. Training works for 50 epochs, then the loss oscillates around a minimum it can never reach because the steps are too large. The optimal learning rate is not a constant. It changes during training. Early on, you want large steps to cover ground quickly. Late in training, you want tiny steps to settle into a sharp minimum. The difference between a 90% accurate model and a 95% accurate model is often just the schedule. Every major model published in the last three years…
Learning Rate Schedules and Warmup: The learning rate is the single most important hyperparameter. Not the architecture. Not the dataset size. Not the…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.