Stephen T’s Blog Spot

You run a solid program, measure the outcome before and after, and the scores barely move. The obvious conclusion is that the program did not work. But there is another explanation that has nothing to do with the program, and it is easy to miss: your measure may have run out of room. If most participants were already near the top of the scale before you started, the instrument cannot show improvement, even when real improvement occurred. The needle did not move because there was nowhere for it to go.

This is a ceiling effect, and it has a mirror image called a floor effect. A ceiling effect happens when a large share of respondents score at or near the maximum, so the scale can no longer tell them apart. Two people who genuinely differ both land at the top, because there is no room above the highest value to separate them. A floor effect is the same failure at the bottom, where everyone piles up at the minimum and the truly struggling look identical to the merely low. In both cases the limit belongs to the instrument, not to the people it is measuring.

The consequences run deeper than a squashed distribution. Real differences at the boundary become invisible, so you cannot distinguish the very good from the excellent, or rank the top performers at all. Change becomes undetectable, which is why a pre-post evaluation built on a saturated measure shows little gain no matter how well the program worked. And the statistics quietly degrade: the distribution turns skewed, the variance shrinks, and correlations with everything else are pulled toward zero. A ceiling does not just hide improvement. It weakens every relationship the variable takes part in, a cousin of the attenuation problem this series has discussed before.

The most consequential mistake is treating the resulting null as a finding about the world. When a measure is compressed against its ceiling, groups that truly differ can look the same, and interventions that truly work can look inert. The report concludes no effect, when the honest conclusion is that the instrument could not have detected one. That is a very different sentence, and it points to a fixable problem rather than a failed program.

Where does the room run out? Usually in the design. A test that is too easy for its takers piles them at the top, and one too hard piles them at the bottom. Coarse scales with only a few points leave little space to move. And a particularly common error in evaluation is repurposing a screening tool, built to detect a problem, as an outcome measure of improvement, so that everyone without the problem sits at the floor with nowhere to fall and no way to show gains. The instrument was calibrated for a different population than the one you are studying.

Catching it is not hard, but it requires looking past the average. Before you trust a null, look at the distribution: what share of responses sit in the very top or very bottom category? If a large fraction are piled against either boundary, your mean is hiding a measurement problem, and a common rule of thumb treats more than about 15 percent at a limit as a warning sign. At design time, the cure is to match the range and difficulty of the instrument to the people you expect and the change you hope to see, to build in headroom at both ends, and to pilot the measure to find where responses accumulate before you rely on it.

For those of us in evaluation, this is a routine trap. Serve a high-functioning population with a measure calibrated to the general public, or use a satisfaction scale where nearly everyone already answers at the top, and you have designed a ceiling into the study before it begins. Before reporting that a program did not move the needle, it is worth asking whether the needle had anywhere to move.

So here is my question. When an evaluation comes back null, do you check whether your measure had room to register the effect, or do you take the flat result at face value?