Random Forests and Bagging
Ensemble concept, RandomForestClassifier, n_estimators, feature importance, OOB score.
Random Forests and Bagging is a free Learn AI with Python lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Learn AI with Python learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
The Problem with Single Trees
A single decision tree has high variance: retraining on slightly different data can produce a very different tree. Ensembles fix this by combining many trees.
Bagging (Bootstrap Aggregating)
Bagging trains many models on different bootstrap samples (random sampling with replacement) of the data, then averages their predictions.
Averaging many high-variance models reduces overall variance without increasing bias much.
from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
bag = BaggingClassifier(
estimator=DecisionTreeClassifier(),
n_estimators=100,
random_state=0,
)
bag.fit(Xtr, ytr)From Bagging to Random Forests
A random forest is bagging of trees plus one extra trick: at each split it considers only a random subset of features.
This decorrelates the trees so their errors cancel out better, improving the ensemble.
RandomForestClassifier Basics
Use RandomForestClassifier. The key parameter n_estimators sets how many trees to build. More trees = more stable but slower.
from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(n_estimators=200, random_state=0)
rf.fit(Xtr, ytr)
print("Accuracy:", rf.score(Xte, yte))Parallel Training with n_jobs
Trees in a forest are independent, so they train in parallel. Set n_jobs=-1 to use all CPU cores and speed up fitting dramatically.
from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(
n_estimators=500,
n_jobs=-1, # use all cores
random_state=0,
)
rf.fit(Xtr, ytr)Out-of-Bag (OOB) Score
Each tree leaves out about one third of samples (the bootstrap did not pick them). These out-of-bag samples form a built-in validation set.
Set oob_score=True to get a free estimate of generalization error without a separate split.
from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(
n_estimators=300,
oob_score=True,
random_state=0,
)
rf.fit(Xtr, ytr)
print("OOB score:", rf.oob_score_)Controlling Tree Depth in the Forest
The same tree parameters apply: max_depth, min_samples_leaf, and max_features (how many features to try per split).
max_features="sqrt" is the common default for classification.
from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(
n_estimators=300,
max_depth=12,
max_features="sqrt",
random_state=0,
)Built-in Feature Importances
Like single trees, forests expose feature_importances_, averaged over all trees. This impurity-based importance is fast but can be biased toward high-cardinality features.
from sklearn.ensemble import RandomForestClassifier
import numpy as np
rf = RandomForestClassifier(n_estimators=200, random_state=0).fit(Xtr, ytr)
order = np.argsort(rf.feature_importances_)[::-1]
for i in order[:5]:
print(i, rf.feature_importances_[i])Permutation Importance
Permutation importance is more reliable: it shuffles one feature column and measures how much the score drops. A big drop means the feature was important.
It is model-agnostic and avoids the impurity bias.
from sklearn.inspection import permutation_importance
result = permutation_importance(
rf, Xte, yte, n_repeats=10, random_state=0, n_jobs=-1
)
print(result.importances_mean)Interpreting Permutation Results
permutation_importance returns importances_mean and importances_std across repeats. Compute it on a held-out set so you measure impact on generalization, not training fit.
import numpy as np
means = result.importances_mean
stds = result.importances_std
for i in np.argsort(means)[::-1]:
print(f"feature {i}: {means[i]:.3f} +/- {stds[i]:.3f}")When to Use Random Forests
Random forests are a strong default: robust, little tuning, handle non-linearities and interactions, and resist overfitting. Downsides are larger memory and slower prediction than a single tree, and less interpretability.
Quick Check
Check your understanding of bagging and forests.
Recap
Recap: Bagging trains models on bootstrap samples and averages them to cut variance. Random forests add random feature subsets per split. Tune with n_estimators, use n_jobs=-1 for speed, get free validation via oob_score, and prefer permutation_importance over impurity-based importance for reliable rankings.
Frequently asked questions
Is the “Random Forests and Bagging” lesson free?
Yes — the full text of “Random Forests and Bagging” is free to read here on the web, and the Learn AI with Python course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Learn AI with Python course, upgrade to CoddyKit PRO.
What will I learn in “Random Forests and Bagging”?
Ensemble concept, RandomForestClassifier, n_estimators, feature importance, OOB score. You practise Learn AI with Python with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Learn AI with Python?
No prior experience is required. Learn AI with Python on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Random Forests and Bagging” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Learn AI with Python lesson?
Yes. Every Learn AI with Python lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Decision Trees: Theory and Implementation
- Random Forests and Bagging
- Gradient Boosting: GBM and XGBoost
- LightGBM and CatBoost