Pourquoi évaluer sur les données d’entraînement est impossible

After training a model, the most tempting thing to do is test it on the same data you used for training. The model will likely score very high — sometimes 100% accuracy — and this feels like success. It is not. This is the most fundamental mistake in machine learning, and it produces results that are completely useless for predicting real-world performance.

Understanding why this fails is not just a technicality — it changes how you think about the entire goal of machine learning. The goal is never to perform well on training data. The goal is always to generalise to new, unseen data.

Consider the difference between a student who memorises every answer in an exam prep book versus one who actually understands the material. The first student scores perfectly on every practice problem but fails when the real exam has slightly different phrasing. The second student may not score perfectly on practice problems but handles new questions confidently.

An ML model that 'memorises' training examples (a deeply overfitted model) behaves exactly like the first student. It achieves perfect training accuracy but fails on new inputs. This phenomenon is called data leakage when it happens during evaluation — you have 'leaked' the answers into the test.

Let us prove this empirically. A decision tree with unlimited depth will memorise every training example perfectly, achieving 100% training accuracy. But its test accuracy will be much lower because it has learned noise rather than the true underlying pattern.

This experiment makes the problem concrete and measurable: training accuracy is meaningless as a performance estimate — it measures memory, not intelligence.

The model's parameters were specifically optimised to minimise error on the training set. This means the training error is guaranteed to be lower than the true generalisation error for any reasonably complex model. The gap between training error and test error is called the optimism of the training error.

The more parameters a model has relative to training examples, the more severe the optimism. A neural network with millions of parameters and only 1,000 training examples can easily achieve 0% training error while its true error rate is 50%. Training error is not a reliable estimate of generalisation — period.

The solution is simple but must be enforced rigorously: reserve a portion of your data before any modelling begins, and never use it for training or for making any decisions that influence the model.

This held-out test set is your honest estimate of generalisation performance. Because the model has never seen it, its performance on this set is the best available estimate of how it will perform on future data. The test set is a one-time-use measurement instrument — using it multiple times to tune your model invalidates it.

Using test data for final evaluation is the obvious form of data leakage. There are subtler forms that are equally destructive:

• Preprocessing leakage: fitting a scaler on the full dataset before splitting, then scaling train and test — the scaler has 'seen' test data statistics.
• Feature leakage: including a feature that is derived from the target variable (e.g., a 'diagnosis confirmed' flag that is only set when a patient is actually sick).
• Time leakage: using future data to predict past events (e.g., including Q3 sales data when predicting Q1 outcomes).

When you use the test set to make decisions — like choosing between two models or selecting a threshold — it is no longer a clean estimate of generalisation. To keep the test set pristine, introduce a validation set:

• Training set: fit model parameters.
• Validation set: tune hyperparameters, select models, adjust thresholds.
• Test set: final one-time evaluation. Touch it only once, at the very end.

A common split is 70% train / 15% validation / 15% test. Cross-validation provides a more efficient alternative by cycling through validation folds on the training set.

Suppose you train 10 different models, evaluate each on the test set, and choose the one with the highest test accuracy. This process has contaminated the test set — you have used test performance to make a modelling decision. The selected model is optimised for the test set, and its reported accuracy is now overly optimistic.

This is exactly why competitions like Kaggle have a public leaderboard (validation set) and a private leaderboard (true test set revealed only at the end). Teams that overfit the public leaderboard by making many submissions often do poorly on the private leaderboard.

The random_state parameter in train_test_split controls which examples end up in which split. Without it, each run produces a different split, making results difficult to reproduce and compare.

Always set random_state to a fixed integer in all code you share with others or want to reproduce later. Any integer works — the convention is to use 42, 0, or 1 — as long as you document which value you used. The specific integer value does not matter; consistency does.

The one rule that encompasses everything in this lesson: the test set must never influence any decision made during model development. This means:

• No preprocessing fitted on test data.
• No model selected based on test performance.
• No threshold tuned on test performance.
• No feature engineered after seeing test errors.
• Test set touched only once, at the very end, to report final performance.

Following this rule ensures your reported performance is an honest estimate of how the model will behave in production on future data.

In this lesson you learned: training accuracy is always optimistically biased because model parameters were optimised on that data, a held-out test set that never influences any training decision provides an honest generalisation estimate, and data leakage — including preprocessing leakage and test set contamination — produces misleadingly high performance estimates. Next up we master scikit-learn's train_test_split function in depth, covering test size ratios, random seeds, and stratification for imbalanced classification problems.