Zufällige Feature-Auswahl: Der Random-Forest-Trick

A plain bagging ensemble trains each tree on a different bootstrap sample, but all trees can still use every feature when choosing a split. This means the most predictive feature will dominate every tree, making them highly correlated. When correlated models are averaged, variance reduction is limited. Random Forests add one extra trick: at every split, only a random subset of features is considered. This decorrelates the trees and dramatically improves the ensemble's generalisation.

In RandomForestClassifier, the max_features parameter controls how many features are candidates at each split. Common choices are 'sqrt' (square root of the total number of features, the default for classification), 'log2', or an integer/float. For regression (RandomForestRegressor), the default is 1.0 (all features), and 'sqrt' or 0.33 are popular alternatives. Smaller values create more diverse trees at the cost of each tree being slightly weaker individually.

Consider a dataset with one dominant feature that predicts the label far better than any other. Without feature sub-sampling, every tree in the ensemble will split on that feature at the root, producing nearly identical trees. Averaging identical predictions yields the same prediction — no variance reduction. When only sqrt(p) features are candidates at each node, that dominant feature is absent in many splits, forcing trees to discover alternative predictive patterns and become genuinely diverse.

Using RandomForestClassifier from scikit-learn is straightforward. You specify the number of trees with n_estimators, control tree complexity with max_depth, and enable OOB evaluation with oob_score=True. The model trains all trees in parallel when n_jobs=-1. After fitting, feature_importances_ gives you a ranked measure of which input features drove the predictions across all trees.

Random Forests compute feature importance as the average decrease in impurity (Gini or entropy) caused by splits on that feature, weighted by the number of samples that pass through each node, averaged across all trees. This gives a fast, global ranking of predictive power. The importances sum to 1 and are accessible via rf.feature_importances_. However, they can be biased toward high-cardinality features, so treat them as a useful heuristic rather than an absolute ground truth.

A single decision tree trained on the full dataset is highly sensitive to noise in the training data — changing a few examples can drastically alter the tree structure. A random forest, by averaging hundreds of such sensitive trees, creates a prediction surface that is much smoother and more robust. This comparison is one of the clearest demonstrations of how ensemble diversity translates into generalisation improvement.

Each tree in a Random Forest is typically grown deep (low bias, high variance). Bagging then corrects the variance. However, for very large datasets you may limit max_depth or min_samples_leaf to reduce memory and training time. Setting max_depth=None (the default) lets trees grow until all leaves are pure. A shallow forest behaves more like a boosting ensemble: low-variance base learners that struggle to capture complex boundaries. The sweet spot depends on your dataset's noise level.

RandomForestRegressor follows the same algorithm but averages the predicted numeric values from each tree rather than voting on a class label. The default max_features for regression in modern scikit-learn is 1.0, but using 'sqrt' or a fraction often helps. Regression forests work well on tabular data with non-linear interactions and are frequently used for house price prediction, energy consumption forecasting, and other continuous-target problems.

Because each tree sees a different bootstrap sample and different feature subsets, the feature importances computed from different runs or different forests can vary. For more reliable importance estimates, use permutation importance (sklearn.inspection.permutation_importance), which measures how much model accuracy drops when a feature's values are randomly shuffled. Permutation importance works for any black-box model and does not suffer from the high-cardinality bias of impurity-based importances.

The most important hyperparameters in a Random Forest are n_estimators (more is usually better, up to a plateau), max_features (controls decorrelation), max_depth and min_samples_leaf (control bias-variance trade-off per tree). A grid search over max_features and min_samples_leaf is often sufficient because n_estimators can be set high and max_depth=None is a good default. Use OOB score for fast approximate tuning before committing to full cross-validation.

Random Forests are powerful but have notable limitations. They are memory-intensive because every tree must be stored. Inference is slow on very large forests since each prediction requires traversing all trees. They are also not the best for structured tabular data with many irrelevant features, where gradient boosting often wins. Finally, Random Forests lack a natural mechanism to capture sequential or spatial structure — for images or text, deep learning methods outperform them decisively.

Test your understanding of Random Forest feature selection concepts from this lesson.

In this lesson you learned: Random Forests add feature sub-sampling to bagging to decorrelate trees, max_features controls the number of candidate features at each split, and feature importances reveal which inputs drive predictions across the forest. Next up we explore out-of-bag error as a free built-in validation mechanism.