Gradient Boosting: GBM and XGBoost
Boosting vs bagging, GradientBoostingClassifier, XGBClassifier, early stopping.
Boosting vs Bagging
Bagging trains trees in parallel and averages them. Boosting trains trees sequentially, where each new tree corrects the errors of the previous ones.
Boosting reduces bias, often yielding higher accuracy than bagging on structured data.
Sequential Weak Learners
Gradient boosting builds an ensemble of shallow trees (weak learners). Each tree fits the residuals (gradients of the loss) left by the current ensemble.
Adding many small corrections builds a strong overall model.
All lessons in this course
- Decision Trees: Theory and Implementation
- Random Forests and Bagging
- Gradient Boosting: GBM and XGBoost
- LightGBM and CatBoost