Erro fora da amostra: validação gratuita dentro da floresta
Ative oob_score, entenda que cada árvore vê apenas cerca de 63% dos dados e use as amostras OOB como um conjunto de validação não enviesado.
Erro fora da amostra: validação gratuita dentro da floresta é uma aula grátis de Machine Learning Academy no CoddyKit. Esta é a aula 3 de 4. Você pode ler a aula completa abaixo gratuitamente — depois pratica ao vivo no navegador com um editor de código integrado e um tutor de IA 24/7. Faz parte do caminho de aprendizado de Machine Learning Academy, e seu progresso é sincronizado entre a web e o app CoddyKit. O curso de Machine Learning Academy inclui 4 aulas no total.
Partes desta aula ainda não foram traduzidas e aparecem em inglês.
What Is Out-of-Bag Error?
When a Random Forest trains each tree on a bootstrap sample, about 37% of training examples are not included in that sample. These out-of-bag (OOB) examples are invisible to the tree during training, so the tree's prediction on them is an honest, held-out prediction. By aggregating OOB predictions across all trees that excluded a given example, we get a nearly unbiased estimate of the model's generalisation error — completely free, with no need for a separate validation split.
How OOB Prediction Is Computed
For each training example x_i, scikit-learn identifies all trees that did not see x_i in their bootstrap sample. Only those trees cast votes for x_i's prediction. The final OOB prediction for x_i is the majority vote (classification) or average (regression) across those trees. This process happens for every training example, giving us an OOB prediction for the full training set. Comparing these predictions to the true labels yields the OOB error.
Enabling OOB Score in scikit-learn
Set oob_score=True when constructing a RandomForestClassifier or RandomForestRegressor. After calling .fit(), the attribute oob_score_ holds the OOB accuracy (or R² for regression). For detailed OOB predictions per sample, access oob_decision_function_ (class probabilities) or oob_prediction_ (regression). The OOB score is essentially a leave-one-forest-out cross-validation estimate computed for free as a byproduct of training.
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
import numpy as np
X, y = load_breast_cancer(return_X_y=True)
rf = RandomForestClassifier(n_estimators=200, oob_score=True, random_state=42)
rf.fit(X, y)
print('OOB score:', round(rf.oob_score_, 4))
cv_score = cross_val_score(rf, X, y, cv=5).mean()
print('5-fold CV score:', round(cv_score, 4))OOB Error vs Cross-Validation
The OOB error closely approximates cross-validation accuracy (typically within 1-2%) but is computed at zero extra training cost. Cross-validation trains the model multiple times (5 or 10 times for 5-fold or 10-fold), whereas the OOB score uses the single training run. The trade-off: OOB requires enough trees (typically 100+) for every training point to be out-of-bag in several trees. With very small forests, some points may be out-of-bag in too few trees and the OOB estimate becomes noisy.
OOB Score as a Function of n_estimators
As you increase n_estimators, the OOB score stabilises. With very few trees, some training examples may be out-of-bag in only 1-2 trees, making per-example predictions noisy. As the forest grows, each example is averaged over more OOB trees and the estimate converges. Plotting OOB score vs n_estimators is a quick way to find the point of diminishing returns — where adding more trees no longer meaningfully improves the score.
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
X, y = load_breast_cancer(return_X_y=True)
for n in [10, 50, 100, 200, 500]:
rf = RandomForestClassifier(n_estimators=n, oob_score=True, random_state=42)
rf.fit(X, y)
print(f'n_estimators={n:4d}: OOB={rf.oob_score_:.4f}')OOB for Regression Forests
For RandomForestRegressor, enabling oob_score=True returns the R² score on OOB predictions by default. You can also access oob_prediction_ — the actual predicted values for each training point from its OOB trees — and compute any metric you prefer, such as MAE or RMSE. This is handy when R² is not the right metric for your problem (e.g., when you care about the absolute scale of errors).
from sklearn.ensemble import RandomForestRegressor
from sklearn.datasets import fetch_california_housing
from sklearn.metrics import mean_absolute_error
import numpy as np
X, y = fetch_california_housing(return_X_y=True)
rfr = RandomForestRegressor(n_estimators=100, oob_score=True, random_state=42)
rfr.fit(X, y)
print('OOB R²:', round(rfr.oob_score_, 4))
oob_mae = mean_absolute_error(y, rfr.oob_prediction_)
print('OOB MAE:', round(oob_mae, 4))Using OOB for Hyperparameter Tuning
Because computing the OOB score is free, you can use it to rapidly explore hyperparameter settings without setting aside a validation fold. Simply iterate over candidate values, train once per configuration, and compare oob_score_. This is especially useful for quick feature selection (remove features and observe OOB change) or setting max_features and min_samples_leaf. Keep in mind that the OOB estimate may be slightly optimistic for very high-dimensional settings.
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
X, y = load_breast_cancer(return_X_y=True)
results = []
for msl in [1, 3, 5, 10, 20]:
rf = RandomForestClassifier(n_estimators=200, min_samples_leaf=msl, oob_score=True, random_state=42)
rf.fit(X, y)
results.append((msl, round(rf.oob_score_, 4)))
best = max(results, key=lambda x: x[1])
for msl, score in results:
print(f'min_samples_leaf={msl:3d}: OOB={score}')
print('Best:', best)OOB Decision Function for Probability Estimates
After training with oob_score=True, the attribute oob_decision_function_ stores a matrix of shape (n_samples, n_classes) containing the class probabilities derived from OOB predictions. These probabilities can be used to plot calibration curves, ROC curves, or precision-recall curves on the training set itself — without touching the test set. This is powerful for diagnosing model behaviour early in development.
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.metrics import roc_auc_score
X, y = load_breast_cancer(return_X_y=True)
rf = RandomForestClassifier(n_estimators=200, oob_score=True, random_state=42)
rf.fit(X, y)
oob_probs = rf.oob_decision_function_[:, 1] # prob of positive class
auc = roc_auc_score(y, oob_probs)
print('OOB AUC-ROC:', round(auc, 4))Ensuring Enough OOB Coverage
A training example is covered by an OOB tree if that tree's bootstrap sample excluded it. By the law of large numbers, with enough trees, every example will have many OOB trees. However, with very small datasets and few trees, some examples might end up being OOB in only 1-2 trees, leading to noisy individual predictions. A rule of thumb: use at least 100 trees for reliable OOB estimates. You can verify coverage by inspecting whether all entries in oob_decision_function_ are non-zero.
OOB Error in the Full ML Workflow
In a typical Random Forest workflow: (1) train on all available labelled data with oob_score=True, (2) use the OOB score to tune hyperparameters via rapid iteration, (3) once satisfied, retrain with the final hyperparameters (still using OOB for the honest score), and (4) make predictions on the true held-out test set only once. This workflow lets you maximise training data usage — you never have to hold out a validation fold — while still getting a reliable performance estimate during development.
Memory and Speed Considerations
Enabling OOB scoring requires scikit-learn to store additional bookkeeping data during training, which increases memory usage slightly. For very large datasets (millions of rows) with many trees, this overhead can become significant. In such cases, use a small holdout set or a single validation fold instead. Also note that OOB predictions are only available for the training examples — you still need the model to predict on new, unseen test examples in the usual way via .predict().
Quick Check
Test your understanding of Out-of-Bag Error concepts from this lesson.
Lesson Recap
In this lesson you learned: OOB examples are the ~37% of training data excluded from each bootstrap sample, aggregating OOB predictions gives a free honest estimate of generalisation error, and oob_score_ can be used to rapidly tune hyperparameters without a separate validation fold. Next up we explore Voting Ensembles that combine different model types together.
Perguntas Frequentes
A aula “Erro fora da amostra: validação gratuita dentro da floresta” é grátis?
Sim — o texto completo de “Erro fora da amostra: validação gratuita dentro da floresta” é grátis para ler aqui na web. Para praticá-la interativamente (um editor de código integrado e um tutor de IA 24/7) e desbloquear o restante do curso de Machine Learning Academy, atualize para CoddyKit PRO. O curso de Machine Learning Academy inclui 4 aulas no total.
O que vou aprender em “Erro fora da amostra: validação gratuita dentro da floresta”?
Ative oob_score, entenda que cada árvore vê apenas cerca de 63% dos dados e use as amostras OOB como um conjunto de validação não enviesado. Você pratica Machine Learning Academy com código prático que executa diretamente no navegador, e um tutor de IA 24/7 responde suas dúvidas enquanto trabalha na aula.
Preciso ter experiência prévia para começar Machine Learning Academy?
Nenhuma experiência prévia é necessária. Machine Learning Academy no CoddyKit é estruturado para alunos iniciantes até avançados, então você pode começar aqui ou desde o início e aprender no seu ritmo. Esta é a aula 3 de 4.
Quanto tempo leva a aula “Erro fora da amostra: validação gratuita dentro da floresta”?
A maioria das aulas CoddyKit leva cerca de 5–10 minutos. Cada uma é compacta e interativa, então você faz progresso constante e retoma exatamente de onde parou entre web e app.
Posso escrever e executar código nesta aula de Machine Learning Academy?
Sim. Cada aula de Machine Learning Academy inclui um editor de código integrado, então você escreve e executa código real direto no navegador e recebe feedback de IA instantaneamente — nenhuma configuração local necessária.
Todas as aulas deste curso
- Agregação por bootstrap (Bagging) explicada
- Seleção aleatória de atributos: o truque da floresta aleatória
- Erro fora da amostra: validação gratuita dentro da floresta
- Conjuntos por votação: voto rígido vs. voto suave