Boosting Intuition: Sequential Error Correction
Learners will simulate three rounds of gradient boosting by hand on a tiny dataset, fitting each new tree to the residuals of the ensemble so far.
Boosting vs Bagging: A Key Distinction
Bagging trains many models in parallel on random subsets of data and combines them, reducing variance. Boosting trains models sequentially: each new model focuses on correcting the mistakes made by the previous ensemble. While bagging reduces variance, boosting primarily reduces bias — it can turn many weak, underfitting models into a powerful learner. This sequential error-correction strategy is the foundation of gradient boosting, AdaBoost, and XGBoost.
The Idea of Sequential Error Correction
Imagine you are predicting house prices. Your first model predicts everything around the mean and makes large errors. Instead of discarding it, you train a second model specifically on the residuals (the errors: actual − predicted) of the first model. The second model learns the patterns the first missed. A third model then corrects what the second still got wrong. Each new model makes a small improvement on the accumulated ensemble's current error. The final prediction is the sum of all models' predictions.
All lessons in this course
- Boosting Intuition: Sequential Error Correction
- XGBoost: Regularisation, Early Stopping, and Feature Importance
- LightGBM: Leaf-Wise Growth and Speed Advantages
- Key Hyperparameters: Learning Rate, n_estimators, and max_depth