K겹 교차 검증: 누수 없이 분할하기

When you evaluate a model on a single held-out test set, the result depends heavily on which examples happened to end up in the test set. With a small dataset, 80 test examples might be unusually easy or hard, making your accuracy estimate misleading. K-Fold Cross-Validation solves this by averaging accuracy across multiple test folds, giving a more reliable estimate of how the model will perform on new data. It also uses the data more efficiently — every example is used for both training and testing across different folds.

K-Fold CV splits the dataset into k equally sized parts called folds. In each of k rounds, one fold is used as the test fold and the remaining k-1 folds form the training set. The model is trained from scratch on the training folds and evaluated on the test fold. This repeats until every fold has served as the test fold exactly once. The final performance estimate is the mean (and standard deviation) of the k scores. Common values: k=5 or k=10.

scikit-learn's cross_val_score handles the entire K-Fold loop: it creates the folds, trains the model on each training portion, evaluates on the test fold, and returns an array of scores. The cv parameter sets the number of folds. The scoring parameter specifies the metric. You can also pass n_jobs=-1 to train all folds in parallel, dramatically reducing wall-clock time on multi-core machines.

The critical rule of K-Fold CV is that the test fold must never influence the model or any preprocessing step. This means the scaler must be fitted only on the training folds and then applied to the test fold — never fitted on the combined data. Using a Pipeline with cross_val_score automatically enforces this: scikit-learn calls fit on the training portion of the pipeline and predict on the test portion inside each fold, preventing any leakage.

By default, KFold does not shuffle the data and always creates the same splits for the same dataset, making results reproducible. If you set shuffle=True, pass a random_state integer so that the shuffle is deterministic: KFold(n_splits=5, shuffle=True, random_state=42). Shuffling is recommended when the dataset is sorted by class or time order, as sequential splits would create highly unrepresentative folds. Always document the random state in your experiments for reproducibility.

The mean CV score estimates your model's expected performance on new data. The standard deviation tells you how stable that performance is. A mean of 0.93 with std 0.01 is much more trustworthy than 0.93 with std 0.08. High standard deviation indicates that model performance is sensitive to which examples are in the training set — a sign of either a small dataset, high model variance, or an unrepresentative data split. When comparing two models, the one with lower variance (smaller std) is often preferred even if its mean is slightly lower.

The choice of k involves a bias-variance trade-off for the CV estimate itself. k=5: each training set is 80% of the data, fast to compute, slightly high bias. k=10: each training set is 90%, better estimate, more computation. Leave-One-Out CV (LOOCV): k=n, nearly unbiased but extremely slow for large datasets and has high variance between folds. The standard recommendation is k=5 or k=10. For very small datasets (<100 examples), LOOCV gives the most data-efficient estimate.

cross_validate is a more powerful variant that can compute multiple metrics simultaneously and also return training scores and fit/score times. This is useful for comparing train vs test scores (to diagnose overfitting) or computing multiple metrics (accuracy, F1, AUC) in a single CV run instead of running CV separately for each metric.

K-Fold CV gives misleading estimates in several scenarios: (1) Time-series data — shuffling and splitting ignores temporal order, creating leakage where future data trains models used to predict the past; (2) Group data — if multiple rows belong to the same patient or user, splitting them into different folds leaks group-level patterns; (3) Highly imbalanced classes — random splits may put all positive examples in the training set. Use specialised CV variants: TimeSeriesSplit, GroupKFold, and StratifiedKFold for these cases.

A common misconception: K-Fold CV does not produce a deployable model. Each of the k model fits is discarded after evaluation. Cross-validation exists solely to estimate how a model trained with your chosen hyperparameters and pipeline will perform on unseen data. Once you are satisfied with the CV estimate, you retrain your final model on the complete training dataset (all folds combined) with those hyperparameters and deploy that model. The CV score is your honest estimate of its expected performance.

When reporting cross-validation results, always report both the mean and the standard deviation: 0.932 ± 0.012. This conveys not just performance but stability. For publication or stakeholder reports, you can also compute a 95% confidence interval: mean ± 1.96 × std / sqrt(k). When comparing two models, check whether their confidence intervals overlap — overlapping intervals suggest the difference is not statistically meaningful and may just reflect random variation in the data split.

Test your understanding of K-Fold Cross-Validation from this lesson.

In this lesson you learned: K-Fold CV estimates generalisation by averaging scores across k non-overlapping test folds, always use a Pipeline to prevent preprocessing from leaking test fold statistics into training, and CV is an evaluation tool — retrain on all data after selecting hyperparameters. Next up we explore Stratified and Time-Series cross-validation for specialised data types.