What is Warmup?

Why does Warmup matter?

Early gradients and optimizer statistics can be unstable, especially in large-batch or transformer training, so abrupt full-size updates may damage optimization.

Warmup in practice

Define warmup in steps or processed tokens, log the realized curve, and tune it with the batch, optimizer, and total training budget held visible.

What is the common confusion about Warmup?

Warmup is not required for every model and does not make an otherwise unsuitable learning rate safe.

Learn Warmup in the course

Lessons that name Warmup in a title or section

• Learning Rate Schedules and Warmup

  The learning rate is the single most important hyperparameter. Not the architecture. Not the dataset size. Not the activation function. The learning rate. If you tune nothing else, tune this.

  Phase 03: Deep Learning Core

• Cosine LR with Linear Warmup

  The learning-rate schedule is the second most important decision after the loss function. AdamW with a cosine decay and a linear warmup is the modern default for language-model training because it…

  Phase 19: Capstone Projects

• Training Loop and Evaluation

  A loop that does not measure is a loop that lies. This lesson builds the training loop that drives the GPT model: AdamW with weight decay split, a warmup plus cosine learning rate schedule, a…

  Phase 19: Capstone Projects

Covered in Phase 03: Deep Learning Core and Phase 19: Capstone Projects.

Related terms

• Learning Rate ScheduleA policy that changes the optimizer's learning rate as training progresses according to steps, epochs, metrics, or a predefined curve.
• Learning RateA scale factor used by an optimizer to control parameter-update magnitude. Values that are too large can destabilize training; values that…
• Batch SizeThe number of examples whose losses contribute to one gradient estimate before an optimizer update.
• AdamWAn Adam variant that decouples weight decay from the gradient-based parameter update.

Sources

• Accurate, Large Minibatch SGD

More terms in Math & training

• Activation Checkpointing
• Activation Function
• Adam (Optimizer)
• AdamW
• Autograd
• Backpropagation
• Batch Size
• Contrastive Learning
• Cross-Entropy
• Data Augmentation
• DPO (Direct Preference Optimization)
• Dropout
• Eigenvalue
• Epoch
• Fine-tuning
• Gradient
• Gradient Accumulation
• Gradient Clipping
• Gradient Descent
• Hyperparameter
• JAX
• Knowledge Distillation
• Learning Rate
• Learning Rate Schedule
• LoRA (Low-Rank Adaptation)
• Loss Function
• Mixed Precision
• NaN (Not a Number)
• Normalization
• Optimizer
• Overfitting
• QLoRA
• ReLU
• RLHF (Reinforcement Learning from Human Feedback)
• SFT (Supervised Fine-Tuning)
• Softmax
• Stochastic Gradient Descent (SGD)
• Transfer Learning
• Underfitting
• Weight
• Weight Decay

Open the Math & training list in the glossary