Stephen T’s Blog Spot

An underpowered study is usually described as one that might miss a real effect. That is true, and it is the least of the problem. The deeper danger is what happens when a small, noisy study does find something. A statistically significant result from an underpowered study is probably a large overestimate of the true effect, and it has a meaningful chance of pointing in the wrong direction entirely.

The logic is worth walking through slowly, because it is not obvious. Suppose the true effect is real but modest, and your study is small or your measurements are noisy. Your estimate will bounce around the truth from sample to sample, sometimes landing near it, sometimes far. Now impose the significance filter. To clear the threshold, an estimate has to be far from zero. The estimates near the true, modest value do not make it. Only the ones that happened to land far out do. Significance therefore selects, systematically, for the exaggerated draws. The finding is not significant despite being extreme. It is significant because it is extreme.

Andrew Gelman and John Carlin gave these failures names in 2014, and the names are useful. Type M error, the exaggeration ratio, is the factor by which a significant estimate overstates the true effect on average. Type S error is the probability that a significant estimate carries the wrong sign, that you conclude the program helped when it actually harmed, or the reverse. Neither of these is captured by the familiar Type I and Type II errors, which speak only to whether an effect was detected, not to whether the number you report bears any resemblance to reality.

The magnitudes involved are sobering. In one published example, Gelman and Carlin analyzed a study whose design gave it roughly 6 percent power. At that level, a significant result would be expected to overstate the true effect by a factor of nearly ten, and would have about a one-in-four chance of having the wrong sign. Read that again. A quarter of the significant findings from such a design would point in the opposite direction from the truth, and the rest would be wildly inflated. The study passed the significance test, and the number it reported was close to meaningless.

This reframes a lot of familiar advice. Statistical power is usually presented as insurance against missing an effect, so a small study is treated as a modest, cautious thing that simply proves less. It is worse than that. An underpowered study that reports nothing has cost you an answer. An underpowered study that reports something has handed you a number that is probably too large and possibly backwards, wrapped in the authority of statistical significance. This is why the literature is littered with dramatic effects that shrink or vanish on replication. They were never that big. The filter selected the flukes.

So what do you do? Gelman and Carlin’s answer is design analysis. Before you run a study, and even after, ask what effect size is actually plausible given what is known, and then work out what your design would produce if that plausible effect were true. How much would a significant estimate be expected to exaggerate it? What is the chance of a wrong-sign result? Doing this requires an honest, externally informed guess about the effect, which is the hard part, and doing it prospectively is what tells you whether the study is worth running at all. A design that would only ever yield a wildly exaggerated estimate is not a cautious study. It is a machine for producing confident nonsense.

For those of us evaluating programs, the practical rule is simple. Do not treat a significant finding from a small pilot or an underpowered evaluation as a conservative estimate of the effect. Treat it as an upper bound at best, and be prepared for the true effect to be far smaller or the reverse of what you found. Ask what effect size the design could plausibly have detected. If the answer is much larger than anything the program could realistically produce, the study cannot tell you what you want to know, whatever the p-value says.

So here is my question. When a small study reports a significant effect, do you treat that number as your best estimate of the truth, or as the exaggerated survivor of a filter that only lets the extreme results through?