PA: Evaluate calibration as well as ranking
A model can order risks well while systematically overstating or understating their expected outcomes. Ranking and calibration answer different questions. Select diagnostics consistent with the target, exposure and business use.
Worked example or practice scenario
Model A predicts twice the observed claim cost in every broad segment but ranks segments correctly. A ranking statistic may look good while a portfolio pricing estimate is severely overstated. Compare aggregate predicted and observed amounts and inspect segment-level calibration.
Try this next
Construct a table with exposure, observed total, predicted total and the ratio for each segment. Distinguish random fluctuation in a small segment from a stable pattern in a large one. Avoid comparing unweighted averages when the intended result is exposure-weighted.
Reading sources
ActNet editorial guide · October 1, 2026 · Original illustrative examples.