Stratified and Time-Series Cross-Validation

Standard KFold splits data randomly without regard for class distribution or temporal order. This creates two serious problems: (1) for imbalanced datasets, some folds may have very few or no minority class examples, making fold-to-fold variation extreme; (2) for time-ordered data, training on future examples to predict the past creates data leakage that inflates CV scores far beyond real-world performance. Specialised CV strategies solve these problems without sacrificing honest evaluation.

StratifiedKFold ensures that each fold contains approximately the same proportion of each class as the full dataset. For example, if 10% of your data is fraudulent, each fold will have approximately 10% fraud cases. This prevents the scenario where one fold has no fraud cases, making it impossible for the classifier to learn to detect fraud in that round. Use StratifiedKFold whenever you have more than 2 classes or significant class imbalance.

You can verify that StratifiedKFold maintains class ratios by inspecting the target distribution in each fold. For each test fold, the proportion of positive examples should match the overall positive rate. Comparing the class distributions from standard KFold and StratifiedKFold on an imbalanced dataset clearly shows why stratification matters — fold class ratios can differ by 20%+ without stratification.

When you pass cv=5 (an integer) to cross_val_score for classification tasks, scikit-learn automatically uses StratifiedKFold. For regression, it uses plain KFold. This means most users get stratification for free without explicitly constructing a StratifiedKFold object. Only override this default when you need custom split behaviour (e.g., group splits, time-series splits, or a different number of folds with shuffling).

Financial, sensor, weather, and many other real-world datasets are ordered in time. A key property of time-series models: predictions must be made using only past data. If you train a model on data from April and use it to predict January values, you are training on the future — this is called temporal data leakage. Even standard K-Fold CV violates this: when fold 3 is the test set and fold 4 is in the training set, future data predicts the past. TimeSeriesSplit prevents this.

TimeSeriesSplit creates folds where training data always precedes test data chronologically. In each split, the training set expands by including the previous test fold. For example, with 5 splits: split 1 trains on the first 20% and tests on the next 20%, split 2 trains on the first 40% and tests on the next 20%, and so on. This ensures no future information enters the training set and the CV score reflects realistic temporal generalisation.

Pass a TimeSeriesSplit object directly to the cv parameter of cross_val_score. The resulting scores measure how well the model generalises when trained on earlier data and evaluated on later data — which is exactly the real-world use case. Note that the training sets grow in size across folds, so earlier folds have fewer training examples and may show lower scores — this is expected, not a bug.

Some datasets have multiple samples per entity — multiple ECGs per patient, multiple transactions per user, multiple images per scene. If the same patient appears in both training and test folds, the model learns patient-specific patterns rather than general patterns. GroupKFold ensures all samples from the same group (patient, user, scene) are in the same fold, preventing this within-group leakage. Pass group identifiers to the groups parameter of cross_val_score.

TimeSeriesSplit uses an expanding window: the training set grows with each split. An alternative is a sliding window: use only the most recent N days/months for training, discarding older data. Sliding windows are better when the relationship between features and target changes over time (concept drift) and older data is no longer representative. scikit-learn does not provide a sliding window CV out of the box, but you can implement it using a custom CV generator or TimeSeriesSplit with max_train_size.

A simple decision guide: (1) Classification with balanced classes → standard KFold or cross_val_score integer (auto-stratified); (2) Classification with imbalance → StratifiedKFold explicitly; (3) Time-ordered data → TimeSeriesSplit; (4) Multiple samples per entity → GroupKFold; (5) Imbalanced + grouped → StratifiedGroupKFold (scikit-learn 0.24+). Getting the CV strategy wrong can make a poorly generalising model look great or a good model look mediocre — always match the CV method to your real-world deployment scenario.

In real production time-series systems, models are retrained periodically and evaluated on the subsequent period. Walk-forward validation simulates this: train on months 1-6, evaluate on month 7; train on months 1-7, evaluate on month 8; and so on. This is equivalent to TimeSeriesSplit with gap parameter to add a buffer between train end and test start (to avoid leakage from closely adjacent time points). The resulting score sequence also reveals whether model performance is stable over time or degrading.

Test your understanding of Stratified and Time-Series cross-validation from this lesson.

In this lesson you learned: StratifiedKFold preserves class proportions across folds for imbalanced classification, TimeSeriesSplit prevents temporal leakage by always training on past data, and GroupKFold prevents within-group leakage when multiple samples belong to the same entity. Next up we compare Grid Search and Random Search for hyperparameter optimisation.