Deborah Mayo (Virginia Tech): “Severe Testing: A Conjecture Passes a Severe Test Only if a Refutation Would Probably Have Occurred if it’s False”

High-profile failures of replication in the social and biological sciences underwrite a minimal requirement of evidence: If a conjecture is retained when little or nothing has been done that would have refuted it, then it has not passed a severe test. This minimal severe-testing requirement leads to reformulating statistical significance tests (and related methods) to avoid familiar criticisms and abuses. The goal of highly well tested claims differs from that of highly probable ones, explaining why experts so often disagree about statistical reforms.

Viewing statistical inference as severe testing–whether or not you accept it–offers a key to understand and get beyond the statistics wars.