SRM: Cross-validation chooses a model without reusing the final test set
Training error favours flexible models. Validation estimates how a modelling procedure performs on data it did not fit. A final test set should remain outside repeated tuning decisions; otherwise it becomes another training signal.
Worked example or practice scenario
Suppose degree-2 and degree-8 polynomial models have training MSE 12 and 3, but validation MSE 14 and 29. The smaller training error does not support choosing degree 8 for prediction. Repeat a suitable resampling procedure to assess variability in that comparison.
Try this next
Fit scaling and any feature selection using only the training portion within each fold. For time-dependent observations, consider time-aware validation instead of assuming random folds imitate deployment. Explain what decision the validation result is supporting.
Reading sources
ActNet editorial guide · October 1, 2026 · Original illustrative examples.