PA: Leakage can survive an apparently strong validation score
Leakage occurs when a model uses information unavailable at prediction time or when preprocessing allows held-out data to influence the fit. A random split does not fix a variable that was created after the outcome.
Worked example or practice scenario
A claim-cost model uses a case-reserve value recorded after an investigation. If the prediction is intended at policy inception, that field is unavailable. Removing it can worsen measured performance while making the model relevant to its actual use.
Try this next
For every feature, record its creation date and availability at the decision time. Fit imputation and feature selection within the training process. Explain how a proposed time split or exclusion addresses the specific leakage mechanism rather than claiming it removes every bias.
Reading sources
ActNet editorial guide · October 1, 2026 · Original illustrative examples.