Ensemble Methods
A group of weak learners, combined correctly, becomes a strong learner. This is not a metaphor. It is a theorem. Language: Python Implement AdaBoost and gradient boosting from scratch and explain how boosting sequentially reduces bias. Build a bagging ensemble and demonstrate how averaging decorrelated models reduces variance without increasing bias. Compare bagging, boosting, and stacking in terms of what error component each method targets. Evaluate ensemble diversity and explain why majority voting accuracy improves with more independent weak learners. A single decision tree is fast to train and easy to interpret, but it overfits. A single linear model underfits on complex boundaries. You could spend days engineering the perfect model architecture. Or you could combine a bunch of imperfect models and get something better than any of them individually. Ensemble methods do exactly this. They are the most reliable technique for winning Kaggle competitions on tabular data, they power most production ML systems, and they illustrate the bias-variance tradeoff in action. Bagging reduces variance. Boosting reduces bias. Stacking learns which models to trust on which inputs. Suppose you have N independent classifiers, each with accuracy p > 0.5. The majority vote has accuracy: For 21 classifiers each with 60% accuracy, majority vote accuracy is about 74%. With 101 classifiers, it rises to 84%. The errors cancel out when the…
Ensemble Methods: A group of weak learners, combined correctly, becomes a strong learner. This is not a metaphor. It is a theorem.
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.