Regularization
Your model gets 99% on training data and 60% on test data. It memorized instead of learning. Regularization is the tax you impose on complexity to force generalization. Implement dropout with inverted scaling, L2 weight decay, batch normalization, layer normalization, and RMSNorm from scratch. Measure the train-test accuracy gap and diagnose overfitting using regularization experiments. Explain why transformers use LayerNorm instead of BatchNorm and why modern LLMs prefer RMSNorm. Apply the correct combination of regularization techniques based on the severity of overfitting. A neural network with enough parameters can memorize any dataset. This is not a hypothetical -- Zhang et al. (2017) proved it by training standard networks on ImageNet with random labels. The networks reached near-zero training loss on completely random label assignments. They memorized a million random input-output pairs with no pattern to learn. Training loss was perfect. Test accuracy was zero. This is the overfitting problem, and it gets worse as models get larger. GPT-3 has 175 billion parameters. The training set has about 500 billion tokens. With that many parameters, the model has enough capacity to memorize significant chunks of the training data verbatim. Without regularization, it would just regurgitate training examples instead of learning generalizable patterns. The gap between training performance and test performance is the overfitting gap. Every technique in this lesson attacks that…
Regularization: Your model gets 99% on training data and 60% on test data. It memorized instead of learning. Regularization is the tax you impose on…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.