Bias-Variance Tradeoff
Every model error comes from one of three sources: bias, variance, or noise. You can only control the first two. Language: Python Derive the bias-variance decomposition of expected prediction error and explain the role of irreducible noise. Diagnose whether a model suffers from high bias or high variance using training and test error patterns. Explain how regularization techniques (L1, L2, dropout, early stopping) trade bias for variance. Implement experiments that visualize the bias-variance tradeoff across models of increasing complexity. You trained a model. It has some error on test data. Where does that error come from? If your model is too simple (linear regression on a curved dataset), it will consistently miss the true pattern. That is bias. If your model is too complex (degree-20 polynomial on 15 data points), it will fit the training data perfectly but give wildly different predictions on new data. That is variance. You cannot minimize both at the same time for a fixed model capacity. Push bias down and variance goes up. Push variance down and bias goes up. Understanding this tradeoff is the single most useful diagnostic skill in machine learning. It tells you whether to make your model more complex or less complex, whether to get more data or engineer better features, whether to regularize more or less. Bias measures how far…
Bias-Variance Tradeoff: Every model error comes from one of three sources: bias, variance, or noise. You can only control the first two.
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.