Boosting Intuition: Sequential Error Correction

Bagging trains many models in parallel on random subsets of data and combines them, reducing variance. Boosting trains models sequentially: each new model focuses on correcting the mistakes made by the previous ensemble. While bagging reduces variance, boosting primarily reduces bias — it can turn many weak, underfitting models into a powerful learner. This sequential error-correction strategy is the foundation of gradient boosting, AdaBoost, and XGBoost.

Imagine you are predicting house prices. Your first model predicts everything around the mean and makes large errors. Instead of discarding it, you train a second model specifically on the residuals (the errors: actual − predicted) of the first model. The second model learns the patterns the first missed. A third model then corrects what the second still got wrong. Each new model makes a small improvement on the accumulated ensemble's current error. The final prediction is the sum of all models' predictions.

Consider a tiny regression dataset with 5 examples. Start with a model that predicts the mean. Compute residuals. Train a tree on residuals. Add a fraction of its predictions (the learning rate) to the current ensemble. Compute new residuals. Repeat. Each round, the residuals should shrink if the trees capture some pattern. After enough rounds, the residuals approach zero for the training set. This is the exact algorithm gradient boosting implements for regression with mean squared error loss.

The learning rate (often called shrinkage) scales down each tree's contribution before adding it to the ensemble. Instead of fully correcting the residual in one step (learning rate = 1.0), boosting takes a small step (e.g., 0.1) and lets subsequent trees correct what remains. A smaller learning rate requires more trees to achieve the same fit but generalises better because no single tree dominates. The trade-off: very small learning rates need very large n_estimators, increasing training time.

Gradient boosting is named after the connection to gradient descent. In standard gradient descent, we update parameters by moving in the direction that reduces the loss. In gradient boosting, we update the prediction function by fitting a new tree to the negative gradient of the loss with respect to the current predictions. For mean squared error loss, the negative gradient is simply the residual (actual − predicted). For other losses (log-loss, absolute error), it computes the appropriate pseudo-residuals.

scikit-learn provides GradientBoostingClassifier which implements the original Friedman gradient boosting algorithm. Key parameters: n_estimators (number of trees), learning_rate (shrinkage), max_depth (depth of each tree — shallow trees are preferred, typically 3-5), and subsample (fraction of training data for each tree, adding stochastic element that improves generalisation). The combination of small learning rate, many shallow trees, and subsampling is the standard recipe.

Unlike bagging, boosting can overfit with enough estimators, especially at a high learning rate. As you add more rounds, the training loss keeps decreasing but the validation loss eventually rises. The solution is: (1) use early stopping — monitor validation loss and stop when it stops improving; (2) use a small learning rate (0.01-0.1) with many trees rather than a large rate with few; (3) regularise individual trees with shallow depth and minimum samples per leaf. These techniques are all exposed in XGBoost and LightGBM's rich hyperparameter sets.

AdaBoost (Adaptive Boosting) was the first practical boosting algorithm. Instead of fitting trees on residuals, it assigns sample weights: misclassified examples get higher weight in the next round, so subsequent trees focus on the hard cases. Each tree's contribution to the final vote is also weighted by its accuracy — better trees get more say. AdaBoost is implemented in scikit-learn as AdaBoostClassifier and is historically important, but gradient boosting (and XGBoost) has largely superseded it in practice.

Setting subsample < 1.0 in gradient boosting introduces randomness: each tree is trained on a random subset of the training data (without replacement). This stochastic element reduces correlation between trees and often improves generalisation — much like the dropout idea in neural networks. A typical value is subsample=0.8. When subsample is less than 1, scikit-learn also computes an out-of-bag estimate of the improvement at each round, accessible via oob_improvement_.

Boosting typically outperforms random forests on clean tabular data where the signal-to-noise ratio is high and there are complex feature interactions. The sequential error-correction mechanism lets boosting squeeze out almost all predictable signal. However, boosting is more sensitive to noisy data and outliers — since each round focuses on hard examples, noise can pull the model in wrong directions. Random forests are more robust to label noise. For noisy, messy real-world data, the choice depends on the specific dataset.

Gradient boosting is equally powerful for regression. GradientBoostingRegressor minimises a regression loss by default (squared error), but also supports absolute error (loss='absolute_error', robust to outliers) and Huber loss (loss='huber', combining squared for small errors and absolute for large ones). For house price prediction, energy forecasting, and similar tasks, gradient boosting consistently outperforms linear regression and often matches or beats neural networks on tabular data without the need for GPU hardware.

Test your understanding of Boosting and Sequential Error Correction from this lesson.

In this lesson you learned: boosting trains models sequentially, each correcting the residual errors of the previous ensemble, the learning rate scales each tree's contribution and controls overfitting, and gradient boosting minimises a differentiable loss by fitting trees to the negative gradient. Next up we explore XGBoost's regularisation and early stopping features.