Chain Rule & Automatic Differentiation
The chain rule is the engine behind every neural network that learns. Language: Python Build a minimal autograd engine (Value class) that records operations and computes gradients via reverse-mode autodiff. Implement forward and backward passes through a computation graph using topological sort. Construct and train a multi-layer perceptron on XOR using only the from-scratch autograd engine. Verify autodiff correctness using gradient checking against numerical finite differences. You can compute derivatives of simple functions. But a neural network is not a simple function. It is hundreds of functions composed together: matrix multiply, add bias, apply activation, matrix multiply again, softmax, cross-entropy loss. The output is a function of a function of a function. To train the network, you need the gradient of the loss with respect to every single weight. Doing this by hand is impossible for millions of parameters. Doing it numerically (finite differences) is too slow. The chain rule gives you the math. Automatic differentiation gives you the algorithm. Together they let you compute exact gradients through arbitrary compositions of functions in time proportional to a single forward pass. This is how PyTorch, TensorFlow, and JAX work. You will build a miniature version from scratch. If y = f(g(x)), the derivative of y with respect to x is: Multiply the derivatives along the chain. Each link contributes its local…
Chain Rule & Automatic Differentiation: The chain rule is the engine behind every neural network that learns. Language: Python
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.