Nested Cross-Validation: Selecting and Evaluating Simultaneously
Learners will structure an outer CV loop for evaluation and an inner loop for hyperparameter selection to get an unbiased estimate of the tuned model's true performance.
Nested Cross-Validation: Selecting and Evaluating Simultaneously is a free Machine Learning Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Machine Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
The Problem of Evaluation Bias
When you use the same data for both hyperparameter selection and performance evaluation, you introduce an optimistic bias. Even if you use cross-validation for selection and a separate test set for evaluation, if you repeat the selection process multiple times (trying different grids, different models), you are implicitly using the test set information. With a single held-out test set, random chance may cause a lucky hyperparameter combination to look better than it truly is. Nested cross-validation provides an unbiased performance estimate while still selecting hyperparameters.
The Two-Loop Structure of Nested CV
Nested CV uses two nested loops: (1) an outer loop for performance evaluation — it creates multiple train/test splits, and the test split is used only to evaluate the final selected model; (2) an inner loop for hyperparameter selection — within each outer training fold, a second CV (or grid search) is run to select the best hyperparameters using only the outer training data. The outer test fold never participates in training or selection. Averaging the outer scores gives the true, unbiased generalisation estimate.
Implementing Nested CV with GridSearchCV
The inner loop is a GridSearchCV object. The outer loop is cross_val_score with the GridSearchCV object as the estimator. cross_val_score calls fit on each outer training fold, which triggers the inner grid search CV, selecting the best hyperparameters within that fold. Then predict is called on the outer test fold using those hyperparameters. The result is an array of scores — one per outer fold — representing honest performance estimates.
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.datasets import load_breast_cancer
import numpy as np
X, y = load_breast_cancer(return_X_y=True)
pipe = Pipeline([('sc', StandardScaler()), ('svc', SVC())])
param_grid = {'svc__C': [0.1, 1, 10], 'svc__gamma': [0.01, 0.1, 1]}
# Inner CV: selects best hyperparameters on outer training fold
inner_cv = GridSearchCV(pipe, param_grid, cv=3, n_jobs=-1)
# Outer CV: gives honest evaluation
outer_scores = cross_val_score(inner_cv, X, y, cv=5, n_jobs=-1)
print('Nested CV scores:', np.round(outer_scores, 4))
print('Unbiased estimate:', round(outer_scores.mean(), 4), '+/-', round(outer_scores.std(), 4))Nested CV vs Non-Nested CV: The Bias Gap
Comparing nested CV scores to non-nested CV scores on the same dataset reveals the optimistic bias of the non-nested approach. The non-nested CV selects hyperparameters based on the same data used for the score estimate, so it will always appear better. The gap between nested and non-nested scores represents how much the hyperparameter selection process overfit to the available data. On small datasets this gap can be substantial (2-5%).
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.datasets import load_breast_cancer
import numpy as np
X, y = load_breast_cancer(return_X_y=True)
pipe = Pipeline([('sc', StandardScaler()), ('svc', SVC())])
param_grid = {'svc__C': [0.1, 1, 10], 'svc__gamma': [0.01, 0.1, 1]}
inner = GridSearchCV(pipe, param_grid, cv=3, n_jobs=-1)
nested_score = cross_val_score(inner, X, y, cv=5).mean()
non_nested = GridSearchCV(pipe, param_grid, cv=5, n_jobs=-1)
non_nested.fit(X, y)
non_nested_score = non_nested.best_score_
print('Non-nested CV score (optimistic):', round(non_nested_score, 4))
print('Nested CV score (unbiased): ', round(nested_score, 4))
print('Optimism gap:', round(non_nested_score - nested_score, 4))When Nested CV Is Essential
Nested CV is most important when: (1) the dataset is small (< 1,000 examples) — test sets are too small for reliable estimates and leakage has more impact; (2) you are comparing multiple algorithm families and selecting the best (model selection adds another layer of overfitting); (3) writing an academic paper that reports unbiased performance. For large datasets or quick prototyping, simple train-validation-test splits or single-level CV with a held-out final test set provide sufficient reliability.
Nested CV for Model Selection
Nested CV extends naturally to selecting among different model families. In each outer fold, the inner search can try multiple algorithms (SVM, random forest, gradient boosting), each with their own hyperparameter grid, and pick the best combination. This gives an honest estimate of the complete model-selection process, not just hyperparameter tuning. This is the rigorous way to answer: 'If I had no prior knowledge, what would I expect my best model to achieve on new data?'
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_breast_cancer
X, y = load_breast_cancer(return_X_y=True)
# Compare SVM vs RF via nested CV
for name, model, grid in [
('SVM', Pipeline([('sc', StandardScaler()), ('svc', SVC())]), {'svc__C': [0.1, 1, 10]}),
('RF', RandomForestClassifier(random_state=42), {'n_estimators': [50, 100, 200]})
]:
inner = GridSearchCV(model, grid, cv=3)
score = cross_val_score(inner, X, y, cv=5).mean()
print(f'{name} nested CV estimate: {round(score, 4)}')Computational Cost of Nested CV
Nested CV is expensive. With an outer 5-fold and inner 3-fold grid search over 9 combinations, you train 5 × 3 × 9 = 135 models. Adding another outer fold or parameter makes cost grow quickly. Cost-reduction strategies: (1) use RandomizedSearchCV in the inner loop; (2) use fewer outer folds (3-fold outer is common); (3) use fewer inner folds; (4) use faster algorithms (LightGBM instead of SVM for large datasets); (5) use n_jobs=-1 at both levels. In production, nested CV is usually a one-time investment during final model validation.
Interpreting Nested CV Results
The outer scores in nested CV represent: 'If I repeated my entire model-development pipeline (CV-based hyperparameter selection) on fresh data, what accuracy would I expect?'. A high mean with low standard deviation is the ideal — it means the selection process is stable and the model generalises consistently. High variance in outer scores suggests that the optimal hyperparameters change significantly across different data samples — a sign of limited data or high model sensitivity.
Extracting Best Parameters from Each Outer Fold
When you pass a GridSearchCV to cross_val_score, it does not expose the best parameters from each outer fold directly. To inspect them, you must implement the outer loop manually using KFold and iterate, fitting the GridSearchCV on each training fold and recording best_params_. This reveals whether the same hyperparameters win in every fold (stable) or different parameters win in different folds (unstable, data-dependent).
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV, KFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.datasets import load_breast_cancer
import numpy as np
X, y = load_breast_cancer(return_X_y=True)
pipe = Pipeline([('sc', StandardScaler()), ('svc', SVC())])
param_grid = {'svc__C': [0.1, 1, 10], 'svc__gamma': [0.01, 0.1, 1]}
outer_cv = KFold(n_splits=3, shuffle=True, random_state=42)
for fold, (tr, te) in enumerate(outer_cv.split(X, y), 1):
inner = GridSearchCV(pipe, param_grid, cv=3)
inner.fit(X[tr], y[tr])
score = inner.score(X[te], y[te])
print(f'Fold {fold}: best={inner.best_params_}, test_acc={score:.4f}')Nested CV in Practice: Final Model
After nested CV confirms your algorithm and hyperparameter selection process will generalise, you retrain the final model. The standard procedure: (1) run nested CV to get an honest performance estimate; (2) run non-nested CV (or a single inner GridSearchCV on all available data) to select the final best hyperparameters; (3) fit the final model with those hyperparameters on all your labelled data; (4) deploy that final model. The nested CV score is your reported metric — it is honest and reproducible.
When Is Nested CV Overkill?
Nested CV is not always necessary. If you have a very large dataset (>100,000 examples), a simple 80/10/10 train/validation/test split works well because the large sample size reduces the variance of each estimate. Similarly, if you are doing rapid prototyping or exploration, non-nested CV is acceptable — just remember to report test set results from a dedicated hold-out that you only look at once. Nested CV matters most in research settings, small-data problems, and competitive benchmarks where every fraction of a percent counts.
Quick Check
Test your understanding of Nested Cross-Validation from this lesson.
Lesson Recap
In this lesson you learned: nested CV separates hyperparameter selection (inner loop) from evaluation (outer loop) to avoid optimistic bias, the outer test fold never participates in training or selection, and nested CV is most critical for small datasets and rigorous academic comparisons. Next up we explore Feature Engineering through log transforms, binning, and interaction features.
Frequently asked questions
Is the “Nested Cross-Validation: Selecting and Evaluating Simultaneously” lesson free?
Yes — the full text of “Nested Cross-Validation: Selecting and Evaluating Simultaneously” is free to read here on the web, and the Machine Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Machine Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “Nested Cross-Validation: Selecting and Evaluating Simultaneously”?
Learners will structure an outer CV loop for evaluation and an inner loop for hyperparameter selection to get an unbiased estimate of the tuned model's true performance. You practise Machine Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Machine Learning Academy?
No prior experience is required. Machine Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Nested Cross-Validation: Selecting and Evaluating Simultaneously” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Machine Learning Academy lesson?
Yes. Every Machine Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- K-Fold Cross-Validation: Splitting Without Leaking
- Stratified and Time-Series Cross-Validation
- Grid Search vs Random Search
- Nested Cross-Validation: Selecting and Evaluating Simultaneously