SRM: Statistical significance does not establish business importance
A p-value addresses a specified statistical hypothesis under the model. It does not measure the size of the effect, prove causation or guarantee useful predictions. Report effect size and uncertainty alongside the test result.
Worked example or practice scenario
A very large dataset can make a fitted 0.2% difference statistically detectable. Whether that difference justifies a business change depends on consequences, uncertainty, stability and the cost of implementation. A large p-value also does not prove that an effect is exactly zero.
Try this next
Write two separate conclusions for a model output: one about the evidence against the null hypothesis and one about the estimated effect. Identify confounding or dependence that could invalidate the model’s assumptions.
Reading sources
ActNet editorial guide · October 1, 2026 · Original illustrative examples.