为什么不能在训练数据上进行评估
您将通过评估一个记忆型模型演示数据泄漏，并了解为什么留出的测试数据对于诚实估计模型性能至关重要
为什么不能在训练数据上进行评估 是 CoddyKit 上的免费 Machine Learning Academy 课时。 这是第 1 节课，共 4 节。 你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。 这是 Machine Learning Academy 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 Machine Learning Academy 课程共包含 4 节课。
本课时的部分内容尚未翻译，以英文显示。
The Evaluation Trap
After training a model, the most tempting thing to do is test it on the same data you used for training. The model will likely score very high — sometimes 100% accuracy — and this feels like success. It is not. This is the most fundamental mistake in machine learning, and it produces results that are completely useless for predicting real-world performance.
Understanding why this fails is not just a technicality — it changes how you think about the entire goal of machine learning. The goal is never to perform well on training data. The goal is always to generalise to new, unseen data.
Memorisation vs Generalisation
Consider the difference between a student who memorises every answer in an exam prep book versus one who actually understands the material. The first student scores perfectly on every practice problem but fails when the real exam has slightly different phrasing. The second student may not score perfectly on practice problems but handles new questions confidently.
An ML model that 'memorises' training examples (a deeply overfitted model) behaves exactly like the first student. It achieves perfect training accuracy but fails on new inputs. This phenomenon is called data leakage when it happens during evaluation — you have 'leaked' the answers into the test.
A Demonstration: Memorisation in Action
Let us prove this empirically. A decision tree with unlimited depth will memorise every training example perfectly, achieving 100% training accuracy. But its test accuracy will be much lower because it has learned noise rather than the true underlying pattern.
This experiment makes the problem concrete and measurable: training accuracy is meaningless as a performance estimate — it measures memory, not intelligence.
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
# Unlimited depth: memorises training data
model = DecisionTreeClassifier() # no max_depth limit
model.fit(X_train, y_train)
train_acc = model.score(X_train, y_train)
test_acc = model.score(X_test, y_test)
print(f'Training accuracy: {train_acc:.3f}') # 1.000 -- perfect memorisation
print(f'Test accuracy: {test_acc:.3f}') # much lower
print(f'Overfit gap: {train_acc - test_acc:.3f}')Why Training Accuracy Is Optimistically Biased
The model's parameters were specifically optimised to minimise error on the training set. This means the training error is guaranteed to be lower than the true generalisation error for any reasonably complex model. The gap between training error and test error is called the optimism of the training error.
The more parameters a model has relative to training examples, the more severe the optimism. A neural network with millions of parameters and only 1,000 training examples can easily achieve 0% training error while its true error rate is 50%. Training error is not a reliable estimate of generalisation — period.
from sklearn.neighbors import KNeighborsClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
import numpy as np
X, y = make_classification(n_samples=500, n_features=20, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)
# KNN with k=1: memorises perfectly (always finds the exact training point)
knn1 = KNeighborsClassifier(n_neighbors=1)
knn1.fit(X_train, y_train)
print(f'k=1 Training accuracy: {knn1.score(X_train, y_train):.3f}') # 1.000
print(f'k=1 Test accuracy: {knn1.score(X_test, y_test):.3f}') # lower
# k=10: generalisers better
knn10 = KNeighborsClassifier(n_neighbors=10)
knn10.fit(X_train, y_train)
print(f'k=10 Test accuracy: {knn10.score(X_test, y_test):.3f}')The Held-Out Test Set: The Solution
The solution is simple but must be enforced rigorously: reserve a portion of your data before any modelling begins, and never use it for training or for making any decisions that influence the model.
This held-out test set is your honest estimate of generalisation performance. Because the model has never seen it, its performance on this set is the best available estimate of how it will perform on future data. The test set is a one-time-use measurement instrument — using it multiple times to tune your model invalidates it.
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_breast_cancer
import numpy as np
X, y = load_breast_cancer(return_X_y=True)
# CORRECT workflow: split BEFORE any analysis
X_train, X_test, y_train, y_test = train_test_split(
X, y,
test_size=0.20, # 20% held out for final evaluation only
random_state=42, # reproducible split
stratify=y # maintain class ratio in both splits
)
print(f'Training set: {X_train.shape[0]} examples')
print(f'Test set: {X_test.shape[0]} examples')
print(f'Train class balance: {y_train.mean():.3f}')
print(f'Test class balance: {y_test.mean():.3f}')Data Leakage: The Subtle Version
Using test data for final evaluation is the obvious form of data leakage. There are subtler forms that are equally destructive:
- Preprocessing leakage: fitting a scaler on the full dataset before splitting, then scaling train and test — the scaler has 'seen' test data statistics.
- Feature leakage: including a feature that is derived from the target variable (e.g., a 'diagnosis confirmed' flag that is only set when a patient is actually sick).
- Time leakage: using future data to predict past events (e.g., including Q3 sales data when predicting Q1 outcomes).
from sklearn.preprocessing import StandardScaler
import numpy as np
# WRONG: fit scaler on full dataset before splitting
X = np.random.randn(1000, 5)
scaler_wrong = StandardScaler()
X_scaled_all = scaler_wrong.fit_transform(X) # leakage! scaler saw test data
# Correct: split first, then fit scaler ONLY on training data
from sklearn.model_selection import train_test_split
X_train, X_test = train_test_split(X, test_size=0.2)
scaler_correct = StandardScaler()
X_train_s = scaler_correct.fit_transform(X_train) # fit on train only
X_test_s = scaler_correct.transform(X_test) # apply to test
print('Correct preprocessing: scaler fitted on training data only.')Three-Way Split: Train, Validation, and Test
When you use the test set to make decisions — like choosing between two models or selecting a threshold — it is no longer a clean estimate of generalisation. To keep the test set pristine, introduce a validation set:
- Training set: fit model parameters.
- Validation set: tune hyperparameters, select models, adjust thresholds.
- Test set: final one-time evaluation. Touch it only once, at the very end.
A common split is 70% train / 15% validation / 15% test. Cross-validation provides a more efficient alternative by cycling through validation folds on the training set.
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_breast_cancer
X, y = load_breast_cancer(return_X_y=True)
# First split off test set (15%)
X_temp, X_test, y_temp, y_test = train_test_split(X, y, test_size=0.15, random_state=42)
# Then split remaining into train and validation
X_train, X_val, y_train, y_val = train_test_split(X_temp, y_temp, test_size=0.176, random_state=42)
# 0.176 of 85% ≈ 15% of the original
print(f'Train: {X_train.shape[0]}')
print(f'Val: {X_val.shape[0]}')
print(f'Test: {X_test.shape[0]}')How Test Set Contamination Inflates Results
Suppose you train 10 different models, evaluate each on the test set, and choose the one with the highest test accuracy. This process has contaminated the test set — you have used test performance to make a modelling decision. The selected model is optimised for the test set, and its reported accuracy is now overly optimistic.
This is exactly why competitions like Kaggle have a public leaderboard (validation set) and a private leaderboard (true test set revealed only at the end). Teams that overfit the public leaderboard by making many submissions often do poorly on the private leaderboard.
from sklearn.datasets import make_classification
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
import numpy as np
X, y = make_classification(n_samples=500, random_state=99)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
# Contamination: picking best max_depth based on test set
best_acc, best_depth = 0, 1
for depth in range(1, 20):
model = DecisionTreeClassifier(max_depth=depth)
model.fit(X_train, y_train)
acc = model.score(X_test, y_test)
if acc > best_acc:
best_acc, best_depth = acc, depth
print(f'Best depth selected by test: {best_depth}, acc: {best_acc:.3f}')
print('This accuracy is now overly optimistic!')Reproducibility: The random_state Parameter
The random_state parameter in train_test_split controls which examples end up in which split. Without it, each run produces a different split, making results difficult to reproduce and compare.
Always set random_state to a fixed integer in all code you share with others or want to reproduce later. Any integer works — the convention is to use 42, 0, or 1 — as long as you document which value you used. The specific integer value does not matter; consistency does.
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_iris
X, y = load_iris(return_X_y=True)
# Without random_state: different results every run
X_train1, X_test1, _, _ = train_test_split(X, y) # no seed
X_train2, X_test2, _, _ = train_test_split(X, y) # no seed
print('Different splits without seed:', not (X_train1 == X_train2).all())
# With random_state: same result every run
X_train3, X_test3, _, _ = train_test_split(X, y, random_state=42)
X_train4, X_test4, _, _ = train_test_split(X, y, random_state=42)
print('Same splits with seed:', (X_train3 == X_train4).all())The Golden Rule of ML Evaluation
The one rule that encompasses everything in this lesson: the test set must never influence any decision made during model development. This means:
- No preprocessing fitted on test data.
- No model selected based on test performance.
- No threshold tuned on test performance.
- No feature engineered after seeing test errors.
- Test set touched only once, at the very end, to report final performance.
Following this rule ensures your reported performance is an honest estimate of how the model will behave in production on future data.
Quick Check
Test your understanding of Machine Learning with Python concepts from this lesson.
Lesson Recap
In this lesson you learned: training accuracy is always optimistically biased because model parameters were optimised on that data, a held-out test set that never influences any training decision provides an honest generalisation estimate, and data leakage — including preprocessing leakage and test set contamination — produces misleadingly high performance estimates. Next up we master scikit-learn's train_test_split function in depth, covering test size ratios, random seeds, and stratification for imbalanced classification problems.
常见问题解答
「为什么不能在训练数据上进行评估」课时是免费的吗？
是的 — 「为什么不能在训练数据上进行评估」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 Machine Learning Academy 课程的其余内容，请升级到 CoddyKit PRO。 Machine Learning Academy 课程共包含 4 节课。
「为什么不能在训练数据上进行评估」这节课中我会学到什么？
您将通过评估一个记忆型模型演示数据泄漏，并了解为什么留出的测试数据对于诚实估计模型性能至关重要 你通过在浏览器中直接运行的动手代码来练习 Machine Learning Academy，全天候 AI 导师会在你学习这节课的过程中回答你的问题。
学习 Machine Learning Academy 需要有经验吗？
无需任何先前经验。CoddyKit 上的 Machine Learning Academy 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。 这是第 1 节课，共 4 节。
「为什么不能在训练数据上进行评估」课时需要多长时间？
大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。
我能在这节 Machine Learning Academy 课中编写并运行代码吗？
能。每节 Machine Learning Academy 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。