Convex Optimization
Convex problems have one valley. Neural networks have millions. Knowing the difference matters. Language: Python Test whether a function is convex using the definition, second derivative, and Hessian criteria. Implement Newton's method and compare its quadratic convergence against gradient descent. Solve constrained optimization problems using Lagrange multipliers and interpret KKT conditions. Explain why neural network loss landscapes are non-convex yet SGD still finds good solutions. Lesson 08 taught you gradient descent, momentum, and Adam. Those optimizers walk downhill on any surface. But they come with no guarantees. Gradient descent on a non-convex landscape might land in a bad local minimum, get stuck on a saddle point, or oscillate forever. You used it anyway because neural networks are non-convex and there is no alternative. But many problems in machine learning are convex. Linear regression, logistic regression, SVMs, LASSO, ridge regression. For these, something stronger exists: optimization with mathematical guarantees. A convex problem has exactly one valley. Any algorithm that walks downhill will reach the global minimum. No restarts needed. No learning rate schedules. No prayer. Understanding convexity does three things. First, it tells you when your problem is easy (convex) versus hard (non-convex). Second, it gives you faster tools like Newton's method for convex problems. Third, it explains concepts that appear throughout ML: regularization as a constraint, duality in SVMs, and…
Convex Optimization: Convex problems have one valley. Neural networks have millions. Knowing the difference matters. Language: Python
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.