Stephen T’s Blog Spot

Over the course of this series I have returned, from several directions, to the same issue. This may be due to a bias that I picked up when I worked at ECRI writing systematic reviews for AHRQ’s Evidence Based Practice Center program. I read an article, authored by Sander Greenland and colleagues, on the worth of the p-value.

P-hacking, where enough analytic choices will eventually produce a significant one. Publication bias, where the literature keeps the significant findings and buries the rest. The gap between an effect that clears a threshold and an effect that matters. And most recently, the discovery that a significant result from an underpowered study is probably exaggerated and might have the wrong sign. Each post attacked a different failure. Taken together they raise a fair question. If the p-value causes this much trouble, what would we do without it?

Start with what a p-value actually is, since much of the trouble begins here. It is the probability of observing data at least as extreme as yours, assuming the null hypothesis and every other assumption in your model are true. That is all. It is not the probability that the null hypothesis is true. It is not the probability your result was a fluke. It does not tell you whether an effect is large, important, or real. The American Statistical Association said as much in its 2016 statement, which was less a critique of the p-value than a catalogue of what people mistakenly believe about it.

Notice that none of my earlier posts actually indicted the p-value itself. They indicted the threshold. The damage comes from dichotomizing a continuous measure of compatibility into significant and not significant, and then treating that binary as a verdict about reality. The threshold is what makes p-hacking worth doing, what tells journals which results to publish, and what filters an underpowered study’s estimates so that only the exaggerated ones survive. The number is a symptom. The line drawn through it is the disease.

The field has taken this seriously. In 2019 the American Statistical Association devoted an entire special issue to a world beyond the 0.05 threshold, and in the same year a comment in Nature by Valentin Amrhein, Sander Greenland, and Blake McShane, endorsed by more than 800 signatories, called for retiring statistical significance. It is worth reading what they actually proposed, because it is more careful than the headlines suggested. They did not call for banning p-values. They called for ending the practice of using them to sort results into two bins, and for treating a p-value as one piece of evidence among many.

We even have a natural experiment. In 2015 the journal Basic and Applied Social Psychology banned null hypothesis significance testing outright, requiring authors to strip p-values, test statistics, and claims about significance before publication. Later assessments of what followed are instructive. Removing the tests did not by itself produce better inference. Authors leaned on descriptive statistics, and without any formal way to express uncertainty, some simply asserted conclusions that the data did not compel. Taking away the crutch does not teach anyone to walk. It can just leave the argument unsupported.

So what fills the space? Not one replacement, which is precisely the point. Report the effect size and an interval around it, and interpret the whole interval rather than checking whether it crosses zero, since the values near its edges are compatible with your data too. Ask what effect size is plausible before you run the study, and what your design would do with it, which is design analysis. State how strong an unmeasured confounder would have to be to overturn your conclusion, which is sensitivity analysis. Bring in prior evidence explicitly, which is what Bayesian methods make natural. And replicate, because a single study, whatever its p-value, was never meant to settle anything.

A world without p-values, then, is not one where the number disappears. It is one where the number stops making decisions. Where a result is described rather than adjudicated, where uncertainty is stated rather than dissolved by a threshold, and where judgment is expected rather than outsourced to a convention. The American Statistical Association’s own summary of this posture is admirably plain: accept uncertainty, and be thoughtful, open, and modest.

That is harder than reading off a threshold, which is exactly why the threshold has survived. A p-value below a line gives a decision-maker something a nuanced interval cannot: permission to stop thinking. Our job, when we hand over findings, is to make the honest version easier to act on than the false certainty it replaces.

So here is my question. If you could not report statistical significance at all, only your estimate, your uncertainty, and your assumptions, would your conclusions change, and would your reader be better served?