Backpropagation from Scratch
Backpropagation is the algorithm that makes learning possible. Without it, neural networks are just expensive random number generators. Implement a Value-based autograd engine that builds a computational graph and computes gradients via topological sort. Derive the backward pass for addition, multiplication, and sigmoid using the chain rule. Train a multi-layer network on XOR and circle classification using only your from-scratch backpropagation engine. Identify the vanishing gradient problem in deep sigmoid networks and explain why gradients shrink exponentially. Your network has a single hidden layer with 768 inputs and 3072 outputs. That's 2,359,296 weights. It made a wrong prediction. Which weights caused the error? Testing each weight individually means 2.3 million forward passes. Backpropagation computes all 2.3 million gradients in a single backward pass. That's not an optimization. That's the difference between trainable and impossible. The naive approach: take one weight, nudge it by a tiny amount, run the forward pass again, measure whether the loss went up or down. That gives you the gradient for that weight. Now do it for every weight in the network. Multiply by thousands of training steps and millions of data points. You'd need geological time to train anything useful. Backpropagation solves this. One forward pass, one backward pass, all gradients computed. The trick is the chain rule from calculus, applied systematically to a computational…
Backpropagation from Scratch: Backpropagation is the algorithm that makes learning possible. Without it, neural networks are just expensive random number…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.