Cosine LR with Linear Warmup
The learning-rate schedule is the second most important decision after the loss function. AdamW with a cosine decay and a linear warmup is the modern default for language-model training because it lets the model see a small effective step size during the brittle first thousand updates, ramps up to a configured peak, and decays smoothly back toward zero. This lesson builds that schedule, plots the curve over training steps, logs gradient norms next to the schedule, and proves the schedule honors warmup, peak, and decay boundaries. Implement an AdamW optimizer wired to a cosine learning-rate schedule with linear warmup. Compute the schedule's exact value at any step without floating-point drift across runs. Log gradient L2 norm side by side with the learning rate so training health is observable. Render the schedule to a text plot the eye can read and a CSV any tool can consume. The first thousand training updates are the loudest. The model's weights are still close to initialization. The optimizer's running second-moment estimate has not stabilised. The gradient norm is large and noisy. If the learning rate is at its peak during these updates the model either diverges outright or settles into a loss plateau it never escapes. The two well-known fixes are gradient clipping, which is the subject of Phase 19 lesson 45, and a…
Cosine LR with Linear Warmup: The learning-rate schedule is the second most important decision after the loss function. AdamW with a cosine decay and a linear…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.