Bootstrap Aggregation (Bagging) Explained

Bootstrap Aggregation, commonly called Bagging, is an ensemble technique that trains multiple models on different random subsets of the training data and combines their predictions. The word bootstrap comes from statistics and means sampling with replacement from your data. By averaging or voting across many diverse models, bagging dramatically reduces the variance of the final prediction without increasing bias.

Sampling with replacement means each bootstrap sample is drawn independently from the full dataset — the same row can appear multiple times in a single sample while other rows may be excluded entirely. For a dataset of N examples, each bootstrap sample also contains N rows. On average, about 63.2% of unique examples appear in any given bootstrap sample, and the remaining ~37% form the out-of-bag set that can be used for validation.

Suppose you have n independent models each with variance σ². If you average their predictions, the variance of the average is σ²/n — it shrinks as you add more models. In practice, models trained on bootstrap samples are correlated (they share the same training distribution), so the reduction is partial but still significant. The key insight is that averaging reduces variance without increasing bias, leading to better generalisation.

You can implement bagging manually by training a list of estimators, each on a different bootstrap sample, then averaging their outputs. This shows exactly what BaggingClassifier does under the hood. Understanding the manual version makes it easier to diagnose problems and extend the technique to custom models.

scikit-learn provides BaggingClassifier that wraps any base estimator and automates the bootstrap sampling and aggregation. You control n_estimators (number of models), max_samples (fraction or absolute count of training samples per model), and max_features (number of features to use). Setting oob_score=True computes a free validation score using the out-of-bag examples for each estimator.

Bagging is not limited to classification. BaggingRegressor applies the same idea: train multiple regressors on bootstrap samples and average their numeric predictions. This is especially valuable when the base learner is a high-variance model like an unpruned decision tree. A single tree memorises noise; the average of many trees smooths it out, reducing the overall test RMSE significantly.

Since ~37% of training examples are excluded from each bootstrap sample, those examples can serve as a validation set for their respective estimator without any additional data split. The OOB score is computed by predicting each training point using only the estimators that did not see it during training. This gives a nearly unbiased estimate of generalisation performance, similar to leave-one-out cross-validation but far cheaper to compute.

Adding more estimators to a bagging ensemble almost always helps (or at worst does not hurt) test performance. Unlike neural network depth, there is no overfitting penalty for adding more trees — variance keeps decreasing while bias stays flat. In practice, performance plateaus after several hundred estimators. The main trade-off is compute time: doubling n_estimators doubles training time. Always check whether the marginal gain justifies the extra cost.

A single complex model (deep tree, high-degree polynomial) achieves low bias but high variance — it fits training data well but fluctuates wildly on unseen data. Bagging tames that variance by averaging across many such models. The combined prediction is much more stable. This is why bagging works best when the base learner is high variance and low bias. Applying bagging to a simple, already-underfitting model (like a linear regression) gives little benefit because the variance was already low.

Each estimator in a bagging ensemble is trained independently, with no data dependencies between them. This makes bagging embarrassingly parallel — all estimators can be trained simultaneously on multiple CPU cores. scikit-learn exposes this via the n_jobs=-1 parameter, which uses all available cores. This is in sharp contrast to boosting methods like XGBoost, where each model depends on the previous one and training is inherently sequential.

Bagging has important limitations. First, it increases compute and memory linearly with the number of estimators. Second, it offers no improvement in bias — if your base model is too simple, bagging many simple models still gives a simple ensemble. Third, bagging makes the model harder to interpret — you lose the single decision tree you could show a stakeholder. Finally, bagging alone does not decorrelate trees if all features are used at every split, which is where Random Forests add the extra trick of feature sub-sampling.

Test your understanding of Bootstrap Aggregation concepts from this lesson.

In this lesson you learned: bootstrap aggregation trains multiple models on random resamples with replacement, averaging predictions reduces variance without increasing bias, and the out-of-bag examples provide free validation for each estimator. Next up we explore the Random Forest trick of sub-sampling features to decorrelate trees further.