Random Feature Selection: The Random Forest Trick
Learners will configure max_features in RandomForestClassifier, observe how feature sub-sampling decorrelates trees, and watch test accuracy rise.
Random Feature Selection: The Random Forest Trick is a free Machine Learning Academy lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Machine Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
From Bagging to Random Forests
A plain bagging ensemble trains each tree on a different bootstrap sample, but all trees can still use every feature when choosing a split. This means the most predictive feature will dominate every tree, making them highly correlated. When correlated models are averaged, variance reduction is limited. Random Forests add one extra trick: at every split, only a random subset of features is considered. This decorrelates the trees and dramatically improves the ensemble's generalisation.
The max_features Parameter
In RandomForestClassifier, the max_features parameter controls how many features are candidates at each split. Common choices are 'sqrt' (square root of the total number of features, the default for classification), 'log2', or an integer/float. For regression (RandomForestRegressor), the default is 1.0 (all features), and 'sqrt' or 0.33 are popular alternatives. Smaller values create more diverse trees at the cost of each tree being slightly weaker individually.
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
import numpy as np
X, y = load_breast_cancer(return_X_y=True)
for mf in ['sqrt', 'log2', 0.5, 1.0]:
rf = RandomForestClassifier(n_estimators=100, max_features=mf, random_state=42)
score = cross_val_score(rf, X, y, cv=5).mean()
print(f'max_features={str(mf):6s}: CV accuracy={score:.4f}')Why Feature Sub-Sampling Decorrelates Trees
Consider a dataset with one dominant feature that predicts the label far better than any other. Without feature sub-sampling, every tree in the ensemble will split on that feature at the root, producing nearly identical trees. Averaging identical predictions yields the same prediction — no variance reduction. When only sqrt(p) features are candidates at each node, that dominant feature is absent in many splits, forcing trees to discover alternative predictive patterns and become genuinely diverse.
Training a RandomForestClassifier
Using RandomForestClassifier from scikit-learn is straightforward. You specify the number of trees with n_estimators, control tree complexity with max_depth, and enable OOB evaluation with oob_score=True. The model trains all trees in parallel when n_jobs=-1. After fitting, feature_importances_ gives you a ranked measure of which input features drove the predictions across all trees.
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
rf = RandomForestClassifier(
n_estimators=200,
max_features='sqrt',
oob_score=True,
n_jobs=-1,
random_state=42
)
rf.fit(X_train, y_train)
print('OOB score:', rf.oob_score_)
print('Test accuracy:', rf.score(X_test, y_test))Feature Importances from Random Forest
Random Forests compute feature importance as the average decrease in impurity (Gini or entropy) caused by splits on that feature, weighted by the number of samples that pass through each node, averaged across all trees. This gives a fast, global ranking of predictive power. The importances sum to 1 and are accessible via rf.feature_importances_. However, they can be biased toward high-cardinality features, so treat them as a useful heuristic rather than an absolute ground truth.
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
data = load_breast_cancer()
X, y = data.data, data.target
feature_names = data.feature_names
rf = RandomForestClassifier(n_estimators=200, random_state=42)
rf.fit(X, y)
importances = pd.Series(rf.feature_importances_, index=feature_names)
print(importances.sort_values(ascending=False).head(5))Comparing Random Forest to a Single Tree
A single decision tree trained on the full dataset is highly sensitive to noise in the training data — changing a few examples can drastically alter the tree structure. A random forest, by averaging hundreds of such sensitive trees, creates a prediction surface that is much smoother and more robust. This comparison is one of the clearest demonstrations of how ensemble diversity translates into generalisation improvement.
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
import numpy as np
X, y = load_breast_cancer(return_X_y=True)
single = DecisionTreeClassifier(random_state=42)
forest = RandomForestClassifier(n_estimators=200, random_state=42)
print('Single tree CV:', np.round(cross_val_score(single, X, y, cv=5), 4))
print('Random forest CV:', np.round(cross_val_score(forest, X, y, cv=5), 4))Controlling Tree Depth in a Forest
Each tree in a Random Forest is typically grown deep (low bias, high variance). Bagging then corrects the variance. However, for very large datasets you may limit max_depth or min_samples_leaf to reduce memory and training time. Setting max_depth=None (the default) lets trees grow until all leaves are pure. A shallow forest behaves more like a boosting ensemble: low-variance base learners that struggle to capture complex boundaries. The sweet spot depends on your dataset's noise level.
Random Forest for Regression
RandomForestRegressor follows the same algorithm but averages the predicted numeric values from each tree rather than voting on a class label. The default max_features for regression in modern scikit-learn is 1.0, but using 'sqrt' or a fraction often helps. Regression forests work well on tabular data with non-linear interactions and are frequently used for house price prediction, energy consumption forecasting, and other continuous-target problems.
from sklearn.ensemble import RandomForestRegressor
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import cross_val_score
import numpy as np
X, y = fetch_california_housing(return_X_y=True)
rfr = RandomForestRegressor(n_estimators=100, max_features='sqrt', n_jobs=-1, random_state=42)
rmse = np.sqrt(-cross_val_score(rfr, X, y, scoring='neg_mean_squared_error', cv=3).mean())
print(f'Random Forest Regression RMSE: {rmse:.4f}')Variance of Feature Importances
Because each tree sees a different bootstrap sample and different feature subsets, the feature importances computed from different runs or different forests can vary. For more reliable importance estimates, use permutation importance (sklearn.inspection.permutation_importance), which measures how much model accuracy drops when a feature's values are randomly shuffled. Permutation importance works for any black-box model and does not suffer from the high-cardinality bias of impurity-based importances.
from sklearn.inspection import permutation_importance
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
import pandas as pd
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
rf = RandomForestClassifier(n_estimators=100, random_state=42).fit(X_train, y_train)
result = permutation_importance(rf, X_test, y_test, n_repeats=10, random_state=42)
print(pd.Series(result.importances_mean, index=load_breast_cancer().feature_names).sort_values(ascending=False).head(5))Hyperparameter Tuning for Random Forests
The most important hyperparameters in a Random Forest are n_estimators (more is usually better, up to a plateau), max_features (controls decorrelation), max_depth and min_samples_leaf (control bias-variance trade-off per tree). A grid search over max_features and min_samples_leaf is often sufficient because n_estimators can be set high and max_depth=None is a good default. Use OOB score for fast approximate tuning before committing to full cross-validation.
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.datasets import load_breast_cancer
X, y = load_breast_cancer(return_X_y=True)
param_grid = {'max_features': ['sqrt', 'log2', 0.5], 'min_samples_leaf': [1, 3, 5]}
grid = GridSearchCV(RandomForestClassifier(n_estimators=100, random_state=42), param_grid, cv=5)
grid.fit(X, y)
print('Best params:', grid.best_params_)
print('Best CV score:', round(grid.best_score_, 4))Limitations of Random Forests
Random Forests are powerful but have notable limitations. They are memory-intensive because every tree must be stored. Inference is slow on very large forests since each prediction requires traversing all trees. They are also not the best for structured tabular data with many irrelevant features, where gradient boosting often wins. Finally, Random Forests lack a natural mechanism to capture sequential or spatial structure — for images or text, deep learning methods outperform them decisively.
Quick Check
Test your understanding of Random Forest feature selection concepts from this lesson.
Lesson Recap
In this lesson you learned: Random Forests add feature sub-sampling to bagging to decorrelate trees, max_features controls the number of candidate features at each split, and feature importances reveal which inputs drive predictions across the forest. Next up we explore out-of-bag error as a free built-in validation mechanism.
Frequently asked questions
Is the “Random Feature Selection: The Random Forest Trick” lesson free?
Yes — the full text of “Random Feature Selection: The Random Forest Trick” is free to read here on the web, and the Machine Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Machine Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “Random Feature Selection: The Random Forest Trick”?
Learners will configure max_features in RandomForestClassifier, observe how feature sub-sampling decorrelates trees, and watch test accuracy rise. You practise Machine Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Machine Learning Academy?
No prior experience is required. Machine Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Random Feature Selection: The Random Forest Trick” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Machine Learning Academy lesson?
Yes. Every Machine Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Bootstrap Aggregation (Bagging) Explained
- Random Feature Selection: The Random Forest Trick
- Out-of-Bag Error: Free Validation Inside the Forest
- Voting Ensembles: Hard Vote vs Soft Vote