Optimizers
Gradient descent tells you which direction to move. It says nothing about how far or how fast. SGD is a compass. Adam is GPS with traffic data. Implement SGD, SGD with momentum, Adam, and AdamW optimizers from scratch in Python. Explain how Adam's bias correction compensates for zero-initialized moment estimates in early training steps. Demonstrate why AdamW produces better generalization than Adam with L2 regularization on the same task. Select the appropriate optimizer and default hyperparameters for transformers, CNNs, GANs, and fine-tuning. You computed the gradients. You know that weight #4,721 should decrease by 0.003 to reduce the loss. But 0.003 in what units? Scaled by what? And should you move the same amount on step 1 as on step 1,000? Vanilla gradient descent applies the same learning rate to every parameter on every step: w = w - lr gradient. This creates three problems that make training neural networks painful in practice. First, oscillation. The loss landscape is rarely shaped like a smooth bowl. It's more like a long, narrow valley. The gradient points across the valley (steep direction), not along it (shallow direction). Gradient descent bounces back and forth across the narrow dimension while making tiny progress along the useful one. You've seen this: loss drops fast then plateaus, not because the model converged but because it's oscillating.…
Optimizers: Gradient descent tells you which direction to move. It says nothing about how far or how fast. SGD is a compass. Adam is GPS with traffic data.
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.