Multi-Layer Networks and Forward Pass
One neuron draws a line. Stack them, and you can draw anything. Build a multi-layer network from scratch with Layer and Network classes that perform a complete forward pass. Trace matrix dimensions through each layer of a network and identify shape mismatches. Explain how stacking nonlinear activations enables a network to learn curved decision boundaries. Solve the XOR problem using a 2-2-1 architecture with hand-tuned sigmoid weights. A single neuron is a line drawer. That's it. One straight line through your data. Every real problem in AI -- image recognition, language understanding, playing Go -- requires curves. Stacking neurons into layers is how you get curves. In 1969, Minsky and Papert proved this limitation was fatal: a single-layer network cannot learn XOR. Not "struggles to learn" -- mathematically cannot. The XOR truth table places [0,1] and [1,0] on one side, [0,0] and [1,1] on the other. No single line separates them. This killed neural network funding for over a decade. The fix was obvious in hindsight: stop using one layer. Stack neurons into layers. Let the first layer carve the input space into new features, and let the second layer combine those features into decisions no single line could make. That stack is the multi-layer network. It is the foundation of every deep learning model in production today. The forward…
Multi-Layer Networks and Forward Pass: One neuron draws a line. Stack them, and you can draw anything. Build a multi-layer network from scratch with Layer and…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.