Optimization
Training a neural network is nothing more than finding the bottom of a valley. Language: Python Implement vanilla gradient descent, SGD with momentum, and Adam from scratch. Compare optimizer convergence on the Rosenbrock function and explain why Adam adapts per-weight learning rates. Distinguish convex from non-convex loss landscapes and explain the role of saddle points in high dimensions. Configure learning rate schedules (step decay, cosine annealing, warmup) for training stability. You have a loss function. It tells you how wrong your model is. You have gradients. They tell you which direction makes the loss worse. Now you need a strategy for walking downhill. The naive approach is simple: move opposite the gradient. Scale the step by some number called the learning rate. Repeat. This is gradient descent, and it works. But "works" has caveats. Too large a learning rate and you overshoot the valley entirely, bouncing between walls. Too small and you crawl toward the answer over thousands of unnecessary steps. Hit a saddle point and you stop moving even though you have not found a minimum. Every optimizer in deep learning is an answer to the same question: how do you get to the bottom of the valley faster and more reliably? Optimization is finding the input values that minimize (or maximize) a function. In machine learning, the function…
Optimization: Training a neural network is nothing more than finding the bottom of a valley. Language: Python Implement vanilla gradient descent, SGD with…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.