入れ子交差検証：選択と評価を同時に行う

When you use the same data for both hyperparameter selection and performance evaluation, you introduce an optimistic bias. Even if you use cross-validation for selection and a separate test set for evaluation, if you repeat the selection process multiple times (trying different grids, different models), you are implicitly using the test set information. With a single held-out test set, random chance may cause a lucky hyperparameter combination to look better than it truly is. Nested cross-validation provides an unbiased performance estimate while still selecting hyperparameters.

Nested CV uses two nested loops: (1) an outer loop for performance evaluation — it creates multiple train/test splits, and the test split is used only to evaluate the final selected model; (2) an inner loop for hyperparameter selection — within each outer training fold, a second CV (or grid search) is run to select the best hyperparameters using only the outer training data. The outer test fold never participates in training or selection. Averaging the outer scores gives the true, unbiased generalisation estimate.

The inner loop is a GridSearchCV object. The outer loop is cross_val_score with the GridSearchCV object as the estimator. cross_val_score calls fit on each outer training fold, which triggers the inner grid search CV, selecting the best hyperparameters within that fold. Then predict is called on the outer test fold using those hyperparameters. The result is an array of scores — one per outer fold — representing honest performance estimates.

Comparing nested CV scores to non-nested CV scores on the same dataset reveals the optimistic bias of the non-nested approach. The non-nested CV selects hyperparameters based on the same data used for the score estimate, so it will always appear better. The gap between nested and non-nested scores represents how much the hyperparameter selection process overfit to the available data. On small datasets this gap can be substantial (2-5%).

Nested CV is most important when: (1) the dataset is small (< 1,000 examples) — test sets are too small for reliable estimates and leakage has more impact; (2) you are comparing multiple algorithm families and selecting the best (model selection adds another layer of overfitting); (3) writing an academic paper that reports unbiased performance. For large datasets or quick prototyping, simple train-validation-test splits or single-level CV with a held-out final test set provide sufficient reliability.

Nested CV extends naturally to selecting among different model families. In each outer fold, the inner search can try multiple algorithms (SVM, random forest, gradient boosting), each with their own hyperparameter grid, and pick the best combination. This gives an honest estimate of the complete model-selection process, not just hyperparameter tuning. This is the rigorous way to answer: 'If I had no prior knowledge, what would I expect my best model to achieve on new data?'

Nested CV is expensive. With an outer 5-fold and inner 3-fold grid search over 9 combinations, you train 5 × 3 × 9 = 135 models. Adding another outer fold or parameter makes cost grow quickly. Cost-reduction strategies: (1) use RandomizedSearchCV in the inner loop; (2) use fewer outer folds (3-fold outer is common); (3) use fewer inner folds; (4) use faster algorithms (LightGBM instead of SVM for large datasets); (5) use n_jobs=-1 at both levels. In production, nested CV is usually a one-time investment during final model validation.

The outer scores in nested CV represent: 'If I repeated my entire model-development pipeline (CV-based hyperparameter selection) on fresh data, what accuracy would I expect?'. A high mean with low standard deviation is the ideal — it means the selection process is stable and the model generalises consistently. High variance in outer scores suggests that the optimal hyperparameters change significantly across different data samples — a sign of limited data or high model sensitivity.

When you pass a GridSearchCV to cross_val_score, it does not expose the best parameters from each outer fold directly. To inspect them, you must implement the outer loop manually using KFold and iterate, fitting the GridSearchCV on each training fold and recording best_params_. This reveals whether the same hyperparameters win in every fold (stable) or different parameters win in different folds (unstable, data-dependent).

After nested CV confirms your algorithm and hyperparameter selection process will generalise, you retrain the final model. The standard procedure: (1) run nested CV to get an honest performance estimate; (2) run non-nested CV (or a single inner GridSearchCV on all available data) to select the final best hyperparameters; (3) fit the final model with those hyperparameters on all your labelled data; (4) deploy that final model. The nested CV score is your reported metric — it is honest and reproducible.

Nested CV is not always necessary. If you have a very large dataset (>100,000 examples), a simple 80/10/10 train/validation/test split works well because the large sample size reduces the variance of each estimate. Similarly, if you are doing rapid prototyping or exploration, non-nested CV is acceptable — just remember to report test set results from a dedicated hold-out that you only look at once. Nested CV matters most in research settings, small-data problems, and competitive benchmarks where every fraction of a percent counts.

In this lesson you learned: nested CV separates hyperparameter selection (inner loop) from evaluation (outer loop) to avoid optimistic bias, the outer test fold never participates in training or selection, and nested CV is most critical for small datasets and rigorous academic comparisons. Next up we explore Feature Engineering through log transforms, binning, and interaction features.