Why You Cannot Evaluate on Training Data
Learners will demonstrate data leakage by evaluating a memorised model and see why held-out test data is essential for honest performance estimates.
Why You Cannot Evaluate on Training Data is a free Machine Learning Academy lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Machine Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
The Evaluation Trap
After training a model, the most tempting thing to do is test it on the same data you used for training. The model will likely score very high — sometimes 100% accuracy — and this feels like success. It is not. This is the most fundamental mistake in machine learning, and it produces results that are completely useless for predicting real-world performance.
Understanding why this fails is not just a technicality — it changes how you think about the entire goal of machine learning. The goal is never to perform well on training data. The goal is always to generalise to new, unseen data.
Memorisation vs Generalisation
Consider the difference between a student who memorises every answer in an exam prep book versus one who actually understands the material. The first student scores perfectly on every practice problem but fails when the real exam has slightly different phrasing. The second student may not score perfectly on practice problems but handles new questions confidently.
An ML model that 'memorises' training examples (a deeply overfitted model) behaves exactly like the first student. It achieves perfect training accuracy but fails on new inputs. This phenomenon is called data leakage when it happens during evaluation — you have 'leaked' the answers into the test.
A Demonstration: Memorisation in Action
Let us prove this empirically. A decision tree with unlimited depth will memorise every training example perfectly, achieving 100% training accuracy. But its test accuracy will be much lower because it has learned noise rather than the true underlying pattern.
This experiment makes the problem concrete and measurable: training accuracy is meaningless as a performance estimate — it measures memory, not intelligence.
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
# Unlimited depth: memorises training data
model = DecisionTreeClassifier() # no max_depth limit
model.fit(X_train, y_train)
train_acc = model.score(X_train, y_train)
test_acc = model.score(X_test, y_test)
print(f'Training accuracy: {train_acc:.3f}') # 1.000 -- perfect memorisation
print(f'Test accuracy: {test_acc:.3f}') # much lower
print(f'Overfit gap: {train_acc - test_acc:.3f}')Why Training Accuracy Is Optimistically Biased
The model's parameters were specifically optimised to minimise error on the training set. This means the training error is guaranteed to be lower than the true generalisation error for any reasonably complex model. The gap between training error and test error is called the optimism of the training error.
The more parameters a model has relative to training examples, the more severe the optimism. A neural network with millions of parameters and only 1,000 training examples can easily achieve 0% training error while its true error rate is 50%. Training error is not a reliable estimate of generalisation — period.
from sklearn.neighbors import KNeighborsClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
import numpy as np
X, y = make_classification(n_samples=500, n_features=20, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)
# KNN with k=1: memorises perfectly (always finds the exact training point)
knn1 = KNeighborsClassifier(n_neighbors=1)
knn1.fit(X_train, y_train)
print(f'k=1 Training accuracy: {knn1.score(X_train, y_train):.3f}') # 1.000
print(f'k=1 Test accuracy: {knn1.score(X_test, y_test):.3f}') # lower
# k=10: generalisers better
knn10 = KNeighborsClassifier(n_neighbors=10)
knn10.fit(X_train, y_train)
print(f'k=10 Test accuracy: {knn10.score(X_test, y_test):.3f}')The Held-Out Test Set: The Solution
The solution is simple but must be enforced rigorously: reserve a portion of your data before any modelling begins, and never use it for training or for making any decisions that influence the model.
This held-out test set is your honest estimate of generalisation performance. Because the model has never seen it, its performance on this set is the best available estimate of how it will perform on future data. The test set is a one-time-use measurement instrument — using it multiple times to tune your model invalidates it.
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_breast_cancer
import numpy as np
X, y = load_breast_cancer(return_X_y=True)
# CORRECT workflow: split BEFORE any analysis
X_train, X_test, y_train, y_test = train_test_split(
X, y,
test_size=0.20, # 20% held out for final evaluation only
random_state=42, # reproducible split
stratify=y # maintain class ratio in both splits
)
print(f'Training set: {X_train.shape[0]} examples')
print(f'Test set: {X_test.shape[0]} examples')
print(f'Train class balance: {y_train.mean():.3f}')
print(f'Test class balance: {y_test.mean():.3f}')Data Leakage: The Subtle Version
Using test data for final evaluation is the obvious form of data leakage. There are subtler forms that are equally destructive:
- Preprocessing leakage: fitting a scaler on the full dataset before splitting, then scaling train and test — the scaler has 'seen' test data statistics.
- Feature leakage: including a feature that is derived from the target variable (e.g., a 'diagnosis confirmed' flag that is only set when a patient is actually sick).
- Time leakage: using future data to predict past events (e.g., including Q3 sales data when predicting Q1 outcomes).
from sklearn.preprocessing import StandardScaler
import numpy as np
# WRONG: fit scaler on full dataset before splitting
X = np.random.randn(1000, 5)
scaler_wrong = StandardScaler()
X_scaled_all = scaler_wrong.fit_transform(X) # leakage! scaler saw test data
# Correct: split first, then fit scaler ONLY on training data
from sklearn.model_selection import train_test_split
X_train, X_test = train_test_split(X, test_size=0.2)
scaler_correct = StandardScaler()
X_train_s = scaler_correct.fit_transform(X_train) # fit on train only
X_test_s = scaler_correct.transform(X_test) # apply to test
print('Correct preprocessing: scaler fitted on training data only.')Three-Way Split: Train, Validation, and Test
When you use the test set to make decisions — like choosing between two models or selecting a threshold — it is no longer a clean estimate of generalisation. To keep the test set pristine, introduce a validation set:
- Training set: fit model parameters.
- Validation set: tune hyperparameters, select models, adjust thresholds.
- Test set: final one-time evaluation. Touch it only once, at the very end.
A common split is 70% train / 15% validation / 15% test. Cross-validation provides a more efficient alternative by cycling through validation folds on the training set.
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_breast_cancer
X, y = load_breast_cancer(return_X_y=True)
# First split off test set (15%)
X_temp, X_test, y_temp, y_test = train_test_split(X, y, test_size=0.15, random_state=42)
# Then split remaining into train and validation
X_train, X_val, y_train, y_val = train_test_split(X_temp, y_temp, test_size=0.176, random_state=42)
# 0.176 of 85% ≈ 15% of the original
print(f'Train: {X_train.shape[0]}')
print(f'Val: {X_val.shape[0]}')
print(f'Test: {X_test.shape[0]}')How Test Set Contamination Inflates Results
Suppose you train 10 different models, evaluate each on the test set, and choose the one with the highest test accuracy. This process has contaminated the test set — you have used test performance to make a modelling decision. The selected model is optimised for the test set, and its reported accuracy is now overly optimistic.
This is exactly why competitions like Kaggle have a public leaderboard (validation set) and a private leaderboard (true test set revealed only at the end). Teams that overfit the public leaderboard by making many submissions often do poorly on the private leaderboard.
from sklearn.datasets import make_classification
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
import numpy as np
X, y = make_classification(n_samples=500, random_state=99)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
# Contamination: picking best max_depth based on test set
best_acc, best_depth = 0, 1
for depth in range(1, 20):
model = DecisionTreeClassifier(max_depth=depth)
model.fit(X_train, y_train)
acc = model.score(X_test, y_test)
if acc > best_acc:
best_acc, best_depth = acc, depth
print(f'Best depth selected by test: {best_depth}, acc: {best_acc:.3f}')
print('This accuracy is now overly optimistic!')Reproducibility: The random_state Parameter
The random_state parameter in train_test_split controls which examples end up in which split. Without it, each run produces a different split, making results difficult to reproduce and compare.
Always set random_state to a fixed integer in all code you share with others or want to reproduce later. Any integer works — the convention is to use 42, 0, or 1 — as long as you document which value you used. The specific integer value does not matter; consistency does.
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_iris
X, y = load_iris(return_X_y=True)
# Without random_state: different results every run
X_train1, X_test1, _, _ = train_test_split(X, y) # no seed
X_train2, X_test2, _, _ = train_test_split(X, y) # no seed
print('Different splits without seed:', not (X_train1 == X_train2).all())
# With random_state: same result every run
X_train3, X_test3, _, _ = train_test_split(X, y, random_state=42)
X_train4, X_test4, _, _ = train_test_split(X, y, random_state=42)
print('Same splits with seed:', (X_train3 == X_train4).all())The Golden Rule of ML Evaluation
The one rule that encompasses everything in this lesson: the test set must never influence any decision made during model development. This means:
- No preprocessing fitted on test data.
- No model selected based on test performance.
- No threshold tuned on test performance.
- No feature engineered after seeing test errors.
- Test set touched only once, at the very end, to report final performance.
Following this rule ensures your reported performance is an honest estimate of how the model will behave in production on future data.
Quick Check
Test your understanding of Machine Learning with Python concepts from this lesson.
Lesson Recap
In this lesson you learned: training accuracy is always optimistically biased because model parameters were optimised on that data, a held-out test set that never influences any training decision provides an honest generalisation estimate, and data leakage — including preprocessing leakage and test set contamination — produces misleadingly high performance estimates. Next up we master scikit-learn's train_test_split function in depth, covering test size ratios, random seeds, and stratification for imbalanced classification problems.
Frequently asked questions
Is the “Why You Cannot Evaluate on Training Data” lesson free?
Yes — the full text of “Why You Cannot Evaluate on Training Data” is free to read here on the web, and the Machine Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Machine Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “Why You Cannot Evaluate on Training Data”?
Learners will demonstrate data leakage by evaluating a memorised model and see why held-out test data is essential for honest performance estimates. You practise Machine Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Machine Learning Academy?
No prior experience is required. Machine Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Why You Cannot Evaluate on Training Data” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Machine Learning Academy lesson?
Yes. Every Machine Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Why You Cannot Evaluate on Training Data
- train_test_split: Ratios, Seeds, and Stratification
- Bias-Variance Trade-off: Underfitting vs Overfitting
- Baseline Models: Always Beat the Dummy Classifier