Learn AI with Python · Lesson

Cross-Validation Strategies

k-fold, stratified k-fold, leave-one-out, time-series split — when to use each.

Cross-Validation Strategies is a free Learn AI with Python lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Learn AI with Python learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Cross-Validation

A single train/test split can be lucky or unlucky. Cross-validation repeats the evaluation on different splits and averages the scores, giving a more reliable estimate of performance.

K-Fold Cross-Validation

KFold splits data into K equal parts. Each fold is used once as the test set while the rest train the model. You get K scores to average.

from sklearn.model_selection import KFold

kf = KFold(n_splits=5, shuffle=True, random_state=0)
for train_idx, test_idx in kf.split(X):
    Xtr, Xte = X[train_idx], X[test_idx]
    ytr, yte = y[train_idx], y[test_idx]
    # train and evaluate here

Stratified K-Fold for Classification

In classification, plain K-Fold can produce folds with imbalanced class ratios. StratifiedKFold preserves the class proportions in every fold.

from sklearn.model_selection import StratifiedKFold

skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
for train_idx, test_idx in skf.split(X, y):
    # each fold keeps the same class balance as the full set
    pass

cross_val_score Shortcut

cross_val_score runs the whole loop for you and returns an array of fold scores. It automatically uses StratifiedKFold for classifiers.

from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier

clf = RandomForestClassifier(random_state=0)
scores = cross_val_score(clf, X, y, cv=5, scoring="accuracy")
print(scores)
print("Mean:", scores.mean(), "Std:", scores.std())

Choosing the Number of Folds

More folds (e.g. 10) give a less biased estimate but cost more compute. 5-fold is a common balance. For very small datasets, leave-one-out uses N folds where each test set is one sample.

from sklearn.model_selection import LeaveOneOut, cross_val_score

loo = LeaveOneOut()
scores = cross_val_score(clf, X, y, cv=loo)
print("Mean:", scores.mean())

TimeSeriesSplit for Temporal Data

For time series you must never train on the future. TimeSeriesSplit always uses past data to train and the next block to test, expanding the training window over folds.

from sklearn.model_selection import TimeSeriesSplit

tscv = TimeSeriesSplit(n_splits=5)
for train_idx, test_idx in tscv.split(X):
    # train_idx always precedes test_idx in time
    print(len(train_idx), len(test_idx))

Why Order Matters in Time Series

Shuffling time-ordered data leaks future information into training and inflates scores. TimeSeriesSplit respects chronological order so your evaluation mirrors real forecasting conditions.

cross_validate for Multiple Metrics

cross_validate is like cross_val_score but returns several metrics at once and can include training scores and fit times.

from sklearn.model_selection import cross_validate

results = cross_validate(
    clf, X, y, cv=5,
    scoring=["accuracy", "f1_macro", "roc_auc_ovr"],
    return_train_score=True,
)
print(results["test_accuracy"])
print(results["test_f1_macro"])

Reading cross_validate Output

The result is a dict with keys like test_<metric>, train_<metric>, fit_time, and score_time. Comparing train vs test scores helps diagnose overfitting.

import numpy as np

gap = results["train_accuracy"].mean() - results["test_accuracy"].mean()
print("Overfit gap:", gap)
print("Avg fit time:", np.mean(results["fit_time"]))

Cross-Validation and Data Leakage

Always fit preprocessing (scaling, encoding) inside the CV loop, not before it. Use a Pipeline so each fold scales using only its training data, preventing leakage.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.model_selection import cross_val_score

pipe = make_pipeline(StandardScaler(), SVC())
scores = cross_val_score(pipe, X, y, cv=5)

Picking the Right Strategy

Use StratifiedKFold for classification, plain KFold for regression, and TimeSeriesSplit for temporal data. Wrap preprocessing in a pipeline and report mean plus standard deviation across folds.

Quick Check

Test your cross-validation knowledge.

Recap

Recap: Cross-validation averages performance over multiple splits. Use KFold for regression, StratifiedKFold for balanced classification, and TimeSeriesSplit for temporal data. cross_val_score gives one metric; cross_validate gives many plus timings. Always preprocess inside a Pipeline to prevent leakage.

Frequently asked questions

Is the “Cross-Validation Strategies” lesson free?

Yes — the full text of “Cross-Validation Strategies” is free to read here on the web, and the Learn AI with Python course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Learn AI with Python course, upgrade to CoddyKit PRO.

What will I learn in “Cross-Validation Strategies”?

k-fold, stratified k-fold, leave-one-out, time-series split — when to use each. You practise Learn AI with Python with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Learn AI with Python?

No prior experience is required. Learn AI with Python on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Cross-Validation Strategies” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Learn AI with Python lesson?

Yes. Every Learn AI with Python lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to Learn AI with Python