Błąd out-of-bag: bezpłatna walidacja wewnątrz lasu
Włącz oob_score, poznaj fakt, że każde drzewo widzi około 63% danych, i użyj próbek OOB jako bezstronnego zbioru walidacyjnego.
Błąd out-of-bag: bezpłatna walidacja wewnątrz lasu to bezpłatna lekcja Machine Learning Academy na CoddyKit. To lekcja 3 z 4. Możesz przeczytać całą lekcję poniżej za darmo — a potem ćwiczyć ją interaktywnie w przeglądarce z wbudowanym edytorem kodu i tutorem AI dostępnym 24/7. To część ścieżki edukacyjnej Machine Learning Academy, a Twój postęp synchronizuje się między webem a aplikacją CoddyKit. Kurs Machine Learning Academy zawiera 4 lekcji w sumie.
Części tej lekcji nie zostały jeszcze przetłumaczone i są wyświetlane po angielsku.
What Is Out-of-Bag Error?
When a Random Forest trains each tree on a bootstrap sample, about 37% of training examples are not included in that sample. These out-of-bag (OOB) examples are invisible to the tree during training, so the tree's prediction on them is an honest, held-out prediction. By aggregating OOB predictions across all trees that excluded a given example, we get a nearly unbiased estimate of the model's generalisation error — completely free, with no need for a separate validation split.
How OOB Prediction Is Computed
For each training example x_i, scikit-learn identifies all trees that did not see x_i in their bootstrap sample. Only those trees cast votes for x_i's prediction. The final OOB prediction for x_i is the majority vote (classification) or average (regression) across those trees. This process happens for every training example, giving us an OOB prediction for the full training set. Comparing these predictions to the true labels yields the OOB error.
Enabling OOB Score in scikit-learn
Set oob_score=True when constructing a RandomForestClassifier or RandomForestRegressor. After calling .fit(), the attribute oob_score_ holds the OOB accuracy (or R² for regression). For detailed OOB predictions per sample, access oob_decision_function_ (class probabilities) or oob_prediction_ (regression). The OOB score is essentially a leave-one-forest-out cross-validation estimate computed for free as a byproduct of training.
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
import numpy as np
X, y = load_breast_cancer(return_X_y=True)
rf = RandomForestClassifier(n_estimators=200, oob_score=True, random_state=42)
rf.fit(X, y)
print('OOB score:', round(rf.oob_score_, 4))
cv_score = cross_val_score(rf, X, y, cv=5).mean()
print('5-fold CV score:', round(cv_score, 4))OOB Error vs Cross-Validation
The OOB error closely approximates cross-validation accuracy (typically within 1-2%) but is computed at zero extra training cost. Cross-validation trains the model multiple times (5 or 10 times for 5-fold or 10-fold), whereas the OOB score uses the single training run. The trade-off: OOB requires enough trees (typically 100+) for every training point to be out-of-bag in several trees. With very small forests, some points may be out-of-bag in too few trees and the OOB estimate becomes noisy.
OOB Score as a Function of n_estimators
As you increase n_estimators, the OOB score stabilises. With very few trees, some training examples may be out-of-bag in only 1-2 trees, making per-example predictions noisy. As the forest grows, each example is averaged over more OOB trees and the estimate converges. Plotting OOB score vs n_estimators is a quick way to find the point of diminishing returns — where adding more trees no longer meaningfully improves the score.
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
X, y = load_breast_cancer(return_X_y=True)
for n in [10, 50, 100, 200, 500]:
rf = RandomForestClassifier(n_estimators=n, oob_score=True, random_state=42)
rf.fit(X, y)
print(f'n_estimators={n:4d}: OOB={rf.oob_score_:.4f}')OOB for Regression Forests
For RandomForestRegressor, enabling oob_score=True returns the R² score on OOB predictions by default. You can also access oob_prediction_ — the actual predicted values for each training point from its OOB trees — and compute any metric you prefer, such as MAE or RMSE. This is handy when R² is not the right metric for your problem (e.g., when you care about the absolute scale of errors).
from sklearn.ensemble import RandomForestRegressor
from sklearn.datasets import fetch_california_housing
from sklearn.metrics import mean_absolute_error
import numpy as np
X, y = fetch_california_housing(return_X_y=True)
rfr = RandomForestRegressor(n_estimators=100, oob_score=True, random_state=42)
rfr.fit(X, y)
print('OOB R²:', round(rfr.oob_score_, 4))
oob_mae = mean_absolute_error(y, rfr.oob_prediction_)
print('OOB MAE:', round(oob_mae, 4))Using OOB for Hyperparameter Tuning
Because computing the OOB score is free, you can use it to rapidly explore hyperparameter settings without setting aside a validation fold. Simply iterate over candidate values, train once per configuration, and compare oob_score_. This is especially useful for quick feature selection (remove features and observe OOB change) or setting max_features and min_samples_leaf. Keep in mind that the OOB estimate may be slightly optimistic for very high-dimensional settings.
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
X, y = load_breast_cancer(return_X_y=True)
results = []
for msl in [1, 3, 5, 10, 20]:
rf = RandomForestClassifier(n_estimators=200, min_samples_leaf=msl, oob_score=True, random_state=42)
rf.fit(X, y)
results.append((msl, round(rf.oob_score_, 4)))
best = max(results, key=lambda x: x[1])
for msl, score in results:
print(f'min_samples_leaf={msl:3d}: OOB={score}')
print('Best:', best)OOB Decision Function for Probability Estimates
After training with oob_score=True, the attribute oob_decision_function_ stores a matrix of shape (n_samples, n_classes) containing the class probabilities derived from OOB predictions. These probabilities can be used to plot calibration curves, ROC curves, or precision-recall curves on the training set itself — without touching the test set. This is powerful for diagnosing model behaviour early in development.
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.metrics import roc_auc_score
X, y = load_breast_cancer(return_X_y=True)
rf = RandomForestClassifier(n_estimators=200, oob_score=True, random_state=42)
rf.fit(X, y)
oob_probs = rf.oob_decision_function_[:, 1] # prob of positive class
auc = roc_auc_score(y, oob_probs)
print('OOB AUC-ROC:', round(auc, 4))Ensuring Enough OOB Coverage
A training example is covered by an OOB tree if that tree's bootstrap sample excluded it. By the law of large numbers, with enough trees, every example will have many OOB trees. However, with very small datasets and few trees, some examples might end up being OOB in only 1-2 trees, leading to noisy individual predictions. A rule of thumb: use at least 100 trees for reliable OOB estimates. You can verify coverage by inspecting whether all entries in oob_decision_function_ are non-zero.
OOB Error in the Full ML Workflow
In a typical Random Forest workflow: (1) train on all available labelled data with oob_score=True, (2) use the OOB score to tune hyperparameters via rapid iteration, (3) once satisfied, retrain with the final hyperparameters (still using OOB for the honest score), and (4) make predictions on the true held-out test set only once. This workflow lets you maximise training data usage — you never have to hold out a validation fold — while still getting a reliable performance estimate during development.
Memory and Speed Considerations
Enabling OOB scoring requires scikit-learn to store additional bookkeeping data during training, which increases memory usage slightly. For very large datasets (millions of rows) with many trees, this overhead can become significant. In such cases, use a small holdout set or a single validation fold instead. Also note that OOB predictions are only available for the training examples — you still need the model to predict on new, unseen test examples in the usual way via .predict().
Quick Check
Test your understanding of Out-of-Bag Error concepts from this lesson.
Lesson Recap
In this lesson you learned: OOB examples are the ~37% of training data excluded from each bootstrap sample, aggregating OOB predictions gives a free honest estimate of generalisation error, and oob_score_ can be used to rapidly tune hyperparameters without a separate validation fold. Next up we explore Voting Ensembles that combine different model types together.
Często zadawane pytania
Czy lekcja „Błąd out-of-bag: bezpłatna walidacja wewnątrz lasu” jest bezpłatna?
Tak — pełny tekst „Błąd out-of-bag: bezpłatna walidacja wewnątrz lasu” jest dostępny za darmo tutaj w sieci. Aby ćwiczyć ją interaktywnie (wbudowany edytor kodu i tutor AI dostępny 24/7) i odblokować resztę kursu Machine Learning Academy, przejdź na CoddyKit PRO. Kurs Machine Learning Academy zawiera 4 lekcji w sumie.
Co nauczysz się w „Błąd out-of-bag: bezpłatna walidacja wewnątrz lasu”?
Włącz oob_score, poznaj fakt, że każde drzewo widzi około 63% danych, i użyj próbek OOB jako bezstronnego zbioru walidacyjnego. Ćwiczysz Machine Learning Academy z praktycznym kodem, który uruchamiasz bezpośrednio w przeglądarce, a tutor AI dostępny 24/7 odpowiada na Twoje pytania podczas pracy nad lekcją.
Czy potrzebuję doświadczenia, aby zacząć Machine Learning Academy?
Nie wymagamy żadnego doświadczenia. Machine Learning Academy w CoddyKit jest strukturyzowany dla początkujących i zaawansowanych użytkowników, więc możesz zacząć tutaj lub od początku i uczyć się w swoim tempie. To lekcja 3 z 4.
Ile czasu zajmuje lekcja „Błąd out-of-bag: bezpłatna walidacja wewnątrz lasu”?
Większość lekcji CoddyKit trwa około 5–10 minut. Każda lekcja to mały, interaktywny krok, dzięki czemu robisz systematyczne postępy i zawsze wracasz dokładnie do tego samego miejsca — na webie i w aplikacji.
Czy mogę pisać i uruchamiać kod w tej lekcji Machine Learning Academy?
Tak. Każda lekcja Machine Learning Academy zawiera wbudowany edytor kodu, więc piszesz i uruchamiasz prawdziwy kod bezpośrednio w przeglądarce i od razu otrzymujesz sprzężenie zwrotne od AI — bez konfiguracji na komputerze.
Wszystkie lekcje w tym kursie
- Agregacja bootstrapowa (bagging) wyjaśniona
- Losowy wybór cech: sztuczka Random Forest
- Błąd out-of-bag: bezpłatna walidacja wewnątrz lasu
- Zespoły głosujące: głosowanie twarde a miękkie