학습, 검증과 테스트 데이터 나누기

If you train and judge a model on the same data, you only learn that it memorized those rows. To trust it, you must split your data into separate roles.

The training set is the data your model actually learns from. Its weights are nudged again and again to fit exactly these examples well.

The validation set is held back during training and checked along the way. It tells you how the model does on data it has never updated on.

The test set is touched once, at the very end. It gives an honest, final score after every choice has already been made.

You tune knobs like learning rate by watching validation results. That quietly leaks the validation set into your choices, so it can no longer be a clean final judge.

Because validation guided your decisions, you need one untouched pile for the truth. Keep the test set sealed until you are completely done.

A typical starting point is roughly 80/10/10: most data for training, smaller slices for validation and test. Adjust based on how much data you have.

PyTorch gives you random_split to carve a dataset into pieces by length. It shuffles which samples land in each set for you.

The split is random, so set a seed if you want the same pieces every run. That keeps experiments comparable across sessions.

With rare classes, a plain random split may leave some out. Stratifying keeps each set's class mix close to the original so scores stay meaningful.

The golden rule is simple: do not let the test set shape any decision. The moment it does, your final number stops being trustworthy.

Three sets, three jobs: training teaches the model, validation guides your tuning, and test gives the final honest score. Keep test sealed until the end. 🎯