Why You Hold Out a Test Set
Estimating performance on unseen data.
Why You Hold Out a Test Set is a free Data Science Academy lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Data Science Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
The Real Question
A model that memorizes your data looks brilliant on it. The real question is how it does on data it has never seen.
Hold Some Data Back
So you set aside part of your data and never train on it. This locked-away slice is your test set, kept for the very end.
Train Here, Judge There
The model learns only from the training set. You then judge it on the untouched test set to see how it truly generalizes. 🎯
Why Not Score on Training
Scoring on the same rows it learned from is like grading a test with the answer key open. That number flatters the model and overstates its skill.
Generalization Is the Goal
You do not care how well it fits old data. You care about generalization: making good predictions on tomorrow's fresh, unseen rows.
Meet Overfitting
When a model nails training data but flops on the test set, it is overfitting. It memorized noise instead of learning the real pattern.
A Common Split
A simple, popular choice is to train on about 80% of rows and test on the remaining 20%. More data to learn, enough left to judge fairly.
Touch It Only Once
The test set is sacred. If you keep peeking and tweaking until the score looks good, you have quietly leaked it into your decisions.
Where the Split Happens
In scikit-learn, one helper does the splitting for you. It shuffles and carves your data into train and test parts in a single call.
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)The Honest Number
The score on the test set is your honest estimate of real-world performance. Trust it more than any glowing training score.
More Than a Formality
Holding out data is not red tape. It is the one habit that stops you from shipping a model that only ever worked on paper.
Quick Check
Why do you keep a separate test set?
Recap
You split data, learn on the train part, and judge on a sacred test set. That untouched slice is your honest read on real-world skill. 🎯
Frequently asked questions
Is the “Why You Hold Out a Test Set” lesson free?
Yes — the full text of “Why You Hold Out a Test Set” is free to read here on the web, and the Data Science Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Data Science Academy course, upgrade to CoddyKit PRO.
What will I learn in “Why You Hold Out a Test Set”?
Estimating performance on unseen data. You practise Data Science Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Data Science Academy?
No prior experience is required. Data Science Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Why You Hold Out a Test Set” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Data Science Academy lesson?
Yes. Every Data Science Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Why You Hold Out a Test Set
- train_test_split Done Right
- K-Fold Cross-Validation
- Stop Data Leakage Before It Starts