K-Fold Cross-Validation: Splitting Without Leaking
Learners will implement 5-fold CV with cross_val_score, understand why the test fold is never used during training, and interpret the mean and standard deviation of CV scores.
K-Fold Cross-Validation: Splitting Without Leaking is a free Machine Learning Academy lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Machine Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Why a Single Train-Test Split Is Risky
When you evaluate a model on a single held-out test set, the result depends heavily on which examples happened to end up in the test set. With a small dataset, 80 test examples might be unusually easy or hard, making your accuracy estimate misleading. K-Fold Cross-Validation solves this by averaging accuracy across multiple test folds, giving a more reliable estimate of how the model will perform on new data. It also uses the data more efficiently — every example is used for both training and testing across different folds.
How K-Fold CV Works Step by Step
K-Fold CV splits the dataset into k equally sized parts called folds. In each of k rounds, one fold is used as the test fold and the remaining k-1 folds form the training set. The model is trained from scratch on the training folds and evaluated on the test fold. This repeats until every fold has served as the test fold exactly once. The final performance estimate is the mean (and standard deviation) of the k scores. Common values: k=5 or k=10.
cross_val_score: One-Line Cross-Validation
scikit-learn's cross_val_score handles the entire K-Fold loop: it creates the folds, trains the model on each training portion, evaluates on the test fold, and returns an array of scores. The cv parameter sets the number of folds. The scoring parameter specifies the metric. You can also pass n_jobs=-1 to train all folds in parallel, dramatically reducing wall-clock time on multi-core machines.
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
import numpy as np
X, y = load_breast_cancer(return_X_y=True)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
scores = cross_val_score(model, X, y, cv=5, scoring='accuracy', n_jobs=-1)
print('Fold scores:', np.round(scores, 4))
print('Mean:', round(scores.mean(), 4), 'Std:', round(scores.std(), 4))The Test Fold Is Never Seen During Training
The critical rule of K-Fold CV is that the test fold must never influence the model or any preprocessing step. This means the scaler must be fitted only on the training folds and then applied to the test fold — never fitted on the combined data. Using a Pipeline with cross_val_score automatically enforces this: scikit-learn calls fit on the training portion of the pipeline and predict on the test portion inside each fold, preventing any leakage.
# WRONG: fitting scaler on all data before CV causes leakage
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer
X, y = load_breast_cancer(return_X_y=True)
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X) # leaks test fold statistics!
wrong_score = cross_val_score(LogisticRegression(max_iter=1000), X_scaled, y, cv=5).mean()
# CORRECT: Pipeline ensures scaler is fitted only on training folds
from sklearn.pipeline import make_pipeline
correct_score = cross_val_score(make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)), X, y, cv=5).mean()
print('Wrong (leaky):', round(wrong_score, 4))
print('Correct (no leak):', round(correct_score, 4))Setting the Random Seed for Reproducibility
By default, KFold does not shuffle the data and always creates the same splits for the same dataset, making results reproducible. If you set shuffle=True, pass a random_state integer so that the shuffle is deterministic: KFold(n_splits=5, shuffle=True, random_state=42). Shuffling is recommended when the dataset is sorted by class or time order, as sequential splits would create highly unrepresentative folds. Always document the random state in your experiments for reproducibility.
from sklearn.model_selection import KFold, cross_val_score
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
import numpy as np
X, y = load_iris(return_X_y=True)
kf = KFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(DecisionTreeClassifier(random_state=42), X, y, cv=kf)
print('Shuffled 5-fold scores:', np.round(scores, 4))
print('Mean:', round(scores.mean(), 4))Interpreting Mean and Standard Deviation
The mean CV score estimates your model's expected performance on new data. The standard deviation tells you how stable that performance is. A mean of 0.93 with std 0.01 is much more trustworthy than 0.93 with std 0.08. High standard deviation indicates that model performance is sensitive to which examples are in the training set — a sign of either a small dataset, high model variance, or an unrepresentative data split. When comparing two models, the one with lower variance (smaller std) is often preferred even if its mean is slightly lower.
Choosing K: 5-Fold vs 10-Fold vs LOOCV
The choice of k involves a bias-variance trade-off for the CV estimate itself. k=5: each training set is 80% of the data, fast to compute, slightly high bias. k=10: each training set is 90%, better estimate, more computation. Leave-One-Out CV (LOOCV): k=n, nearly unbiased but extremely slow for large datasets and has high variance between folds. The standard recommendation is k=5 or k=10. For very small datasets (<100 examples), LOOCV gives the most data-efficient estimate.
from sklearn.model_selection import cross_val_score, LeaveOneOut
from sklearn.neighbors import KNeighborsClassifier
from sklearn.datasets import load_iris
import numpy as np
X, y = load_iris(return_X_y=True)
knn = KNeighborsClassifier(n_neighbors=5)
for k in [5, 10]:
scores = cross_val_score(knn, X, y, cv=k)
print(f'{k}-fold CV: mean={scores.mean():.4f}, std={scores.std():.4f}')
# LOOCV is slow for large datasets; feasible here
loo_scores = cross_val_score(knn, X, y, cv=LeaveOneOut())
print(f'LOOCV: mean={loo_scores.mean():.4f}, std={loo_scores.std():.4f}')cross_validate: Multiple Metrics at Once
cross_validate is a more powerful variant that can compute multiple metrics simultaneously and also return training scores and fit/score times. This is useful for comparing train vs test scores (to diagnose overfitting) or computing multiple metrics (accuracy, F1, AUC) in a single CV run instead of running CV separately for each metric.
from sklearn.model_selection import cross_validate
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
import numpy as np
X, y = load_breast_cancer(return_X_y=True)
rf = RandomForestClassifier(n_estimators=100, random_state=42)
results = cross_validate(rf, X, y, cv=5,
scoring=['accuracy', 'f1', 'roc_auc'],
return_train_score=True)
for metric in ['test_accuracy', 'test_f1', 'test_roc_auc']:
vals = results[metric]
print(f'{metric}: {vals.mean():.4f} (+/- {vals.std():.4f})')When K-Fold CV Can Be Misleading
K-Fold CV gives misleading estimates in several scenarios: (1) Time-series data — shuffling and splitting ignores temporal order, creating leakage where future data trains models used to predict the past; (2) Group data — if multiple rows belong to the same patient or user, splitting them into different folds leaks group-level patterns; (3) Highly imbalanced classes — random splits may put all positive examples in the training set. Use specialised CV variants: TimeSeriesSplit, GroupKFold, and StratifiedKFold for these cases.
Cross-Validation Is Evaluation, Not Training
A common misconception: K-Fold CV does not produce a deployable model. Each of the k model fits is discarded after evaluation. Cross-validation exists solely to estimate how a model trained with your chosen hyperparameters and pipeline will perform on unseen data. Once you are satisfied with the CV estimate, you retrain your final model on the complete training dataset (all folds combined) with those hyperparameters and deploy that model. The CV score is your honest estimate of its expected performance.
Reporting CV Results: Mean and Confidence Interval
When reporting cross-validation results, always report both the mean and the standard deviation: 0.932 ± 0.012. This conveys not just performance but stability. For publication or stakeholder reports, you can also compute a 95% confidence interval: mean ± 1.96 × std / sqrt(k). When comparing two models, check whether their confidence intervals overlap — overlapping intervals suggest the difference is not statistically meaningful and may just reflect random variation in the data split.
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
import numpy as np
X, y = load_breast_cancer(return_X_y=True)
scores = cross_val_score(RandomForestClassifier(n_estimators=100, random_state=42), X, y, cv=10)
mean, std = scores.mean(), scores.std()
ci95 = 1.96 * std / np.sqrt(len(scores))
print(f'CV Mean: {mean:.4f}')
print(f'CV Std: {std:.4f}')
print(f'95% CI: [{mean-ci95:.4f}, {mean+ci95:.4f}]')Quick Check
Test your understanding of K-Fold Cross-Validation from this lesson.
Lesson Recap
In this lesson you learned: K-Fold CV estimates generalisation by averaging scores across k non-overlapping test folds, always use a Pipeline to prevent preprocessing from leaking test fold statistics into training, and CV is an evaluation tool — retrain on all data after selecting hyperparameters. Next up we explore Stratified and Time-Series cross-validation for specialised data types.
Frequently asked questions
Is the “K-Fold Cross-Validation: Splitting Without Leaking” lesson free?
Yes — the full text of “K-Fold Cross-Validation: Splitting Without Leaking” is free to read here on the web, and the Machine Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Machine Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “K-Fold Cross-Validation: Splitting Without Leaking”?
Learners will implement 5-fold CV with cross_val_score, understand why the test fold is never used during training, and interpret the mean and standard deviation of CV scores. You practise Machine Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Machine Learning Academy?
No prior experience is required. Machine Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “K-Fold Cross-Validation: Splitting Without Leaking” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Machine Learning Academy lesson?
Yes. Every Machine Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- K-Fold Cross-Validation: Splitting Without Leaking
- Stratified and Time-Series Cross-Validation
- Grid Search vs Random Search
- Nested Cross-Validation: Selecting and Evaluating Simultaneously