Train/validation/test splits and k-fold cross-validation

Three sets, three jobs

• Training set: the model learns its parameters (weights, tree splits) from it.
• Validation set: used to choose hyperparameters and models: k in k-NN, tree depth, regularisation strength, which features to keep.
• Test set: touched once, at the end, to estimate performance on unseen data. If you keep adjusting the model after looking at test scores, the test set effectively becomes a validation set and the estimate becomes optimistic.

A common starting split is 70/15/15 or 80/10/10. The right ratio depends on how much data you have.

k-fold cross-validation (CV). Split the non-test data into k equal folds, often k = 5 or 10. Train k times, each time holding out a different fold for validation. Report the mean and standard deviation of the k scores.

• Every row is used for validation exactly once, which matters on small data sets.
• It costs k times the training time.
• Nested CV (an inner loop for tuning, an outer loop for estimating) gives an honest estimate when you tune heavily.

When a random split is wrong

• Imbalanced classes → stratified split (StratifiedKFold, or train_test_split(..., stratify=y)). Each fold then keeps the same class proportions, so a fold cannot end up with no fraud cases at all.
• Time series → time-based split (TimeSeriesSplit). Always train on the past and validate on the future. Random shuffling lets the model "see the future" and inflates scores.
• Grouped data → group split (GroupKFold). If one patient, user or device has many rows, keep all of their rows in the same fold. Otherwise the model can recognise the individual instead of learning the pattern.
• Duplicates or near-duplicates must not appear on both sides.

Remember: fit preprocessing (scalers, imputers, encoders) inside each training fold to avoid leakage.