Train, validation and test sets — and data leakage

Ask which set is used to make decisions about the model.

Model answer.

Purpose of each split

• Training set: used to fit the model's parameters (weights).
• Validation set: used to make decisions about the model — choosing hyperparameters (learning rate, tree depth, regularisation strength), comparing algorithms, and early stopping.
• Test set: used once, at the end, to estimate how the final model performs on unseen data.

Why three? Every time you tune against a dataset, your choices adapt to it. If you tuned hyperparameters on the test set, its score would be optimistically biased and no longer an honest estimate. The validation set absorbs that tuning so the test set stays untouched.

Typical splits are 70/15/15 or 80/10/10. With little data, use k-fold cross-validation on the training portion instead of a single validation set, and still hold out a test set.

Data leakage
Leakage is when information that would not be available at prediction time influences training, making evaluation look better than real-world performance.

Common sources:

• Preprocessing before splitting: fitting a scaler, imputer or feature selector on the full dataset lets statistics from the test set leak into training. Fit them on the training data only and apply to validation/test (pipelines help).
• Wrong split for time series: randomly shuffling stock prices or sales data lets the model learn from the future. Split by time — train on the past, validate on later periods.
• Target leakage in features: e.g. predicting loan default using a "collections_agency_assigned" column that is only filled after a default.
• Duplicates or related records across splits: the same patient's scans appearing in both train and test. Split by group (patient, user).

Other good practice: use a stratified split for imbalanced classification so each split keeps the class ratio.

Common follow-up: "Your model gets 99% on test but fails in production — what do you check first?" Look for leakage, then for distribution shift between the training data and live data.