Calculus, Gradients and Convex Optimization

Derivatives, autodiff, GD, momentum, Adam, Newton and convexity tests. From-scratch Python and PyTorch.

Forward and backward through one neuron#

Task	Code
Central difference	`numerical_derivative(f, x, h=1e-7)`
Gradient of many inputs	`numerical_gradient(f, point, h=1e-7)`
Hessian of f(x, y)	`hessian_2d(f, x, y, h=1e-5)`
Check an autodiff gradient	`gradient_check(build_expr, x_val, h=1e-7)`
XOR network, 4 hidden	`model = MLP([2, 4, 1])`
Run an optimizer	`optimize(optimizer, func, grad_func, start)`
Test convexity by sampling	`check_convexity(f, dim)`
Newton steps in 2D	`newtons_method(grad_f, hessian_f, x0)`

GradientDescent(lr=0.001)w = w - lr * gradient

SGDMomentum(lr=0.001, momentum=0.9)v = beta * v + gradient

Adam(lr=0.001, beta1=0.9, beta2=0.999, epsilon=1e-8)two moments per weight

m_hat = [m / (1 - self.beta1 ** self.t) for m in self.m]bias correction

p - self.lr * mh / (vh ** 0.5 + self.epsilon)per-weight step size

adam_history = optimize(Adam(lr=0.01), rosenbrock, rosenbrock_gradient, start)

x = torch.tensor(2.0, requires_grad=True)

y = x ** 2 + 3 * x + 1

y.backward()reverse mode through the recorded graph

print(x.grad)7.0 = 2*x + 3

sgd = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9)

adam = torch.optim.Adam(model.parameters(), lr=0.001)

adamw = torch.optim.AdamW(model.parameters(), lr=0.001, weight_decay=0.01)

scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(adam, T_max=100)

Command	What it does
x^2	2x. MSE loss
wx + b	x for w, 1 for b, w for x. Linear layer
e^x	e^x. Softmax, attention
ln(x)	1/x. Cross-entropy loss
sigmoid	f(x)(1-f(x)). Sigmoid activation
tanh	1 - tanh^2. Classic activation
x^n	n x^(n-1). Used by `__pow__`
sin(x^2)	cos(x^2) * 2x. Chain rule example

Command	What it does
Gradient descent	First derivatives. O(n) memory. Baseline
SGD with momentum	Best final accuracy with more tuning
Adam	Diagonal curvature from gradient moments. Deep learning default
AdamW	Adam with decoupled weight decay. Transformers
Newton's method	Full Hessian. O(n^3) per step. Small convex problems
L-BFGS	Last m gradient differences. Medium convex problems
K-FAC	Kronecker-factored Fisher. Large-batch training

Setting	Default	Note
Adam `lr`	0.001	Start here for most problems
Adam `beta2`	0.999	`beta1` is 0.9. Moment decay rates
Adam `epsilon`	1e-8	Stops division by zero
SGD momentum `lr`	0.01	With momentum 0.9
AdamW `weight_decay`	0.01	Decoupled from the gradient step
Mini-batch size	32-256	Good gradient estimate, moderate noise
Finite difference `h`	1e-7	In `numerical_derivative`
Gradient check difference	< 1e-5	Autodiff against numerical

Symptom	Cause	Fix
Loss grows or becomes nan	Learning rate too large	Reduce the learning rate
Training is too slow	Learning rate too small	Increase it. Add a schedule
Zigzag in a narrow valley	High condition number of the Hessian	Add momentum, or use Newton
Gradient near 0, no progress	Saddle point	Momentum and mini-batch noise
Gradients grow each step	`.grad` adds with `+=`	Set `p.grad = 0.0` before `backward`
Autodiff disagrees with numeric	Bug in a `_backward` rule	Run `gradient_check`
Noisy second derivative	Step h too small	Use `h=1e-4` for both calls
Newton too costly for a big model	O(n^2) memory, O(n^3) inverse	Use L-BFGS or Adam

Why reverse mode for networks?: Many inputs and one loss. One backward pass gives all gradients.
How do I test convexity?: f'' >= 0 in 1D. Hessian positive semidefinite in many dimensions.
Why does SGD work on non-convex losses?: Most critical points are saddle points. Most minima are good.
Why does L1 give sparse weights?: The diamond has corners on the axes.
What is complementary slackness?: $inline$ . Support vectors have $inline$ .

Use a learning rate schedule for runs longer than a few epochs.
Cosine annealing and warmup suit transformers and large models.
Newton's method solves a quadratic in one step.
Gradient checks cost 2 forward passes per parameter. Do not run them in production training.
A Hessian with mixed eigenvalue signs at a critical point marks a saddle.
Flat minima generalize better than sharp minima.