Stephen T’s Blog Spot

You field a survey, score it, and compare two groups. Men score higher than women on the scale, or one site outperforms another, or the average climbs after the program. The natural next move is to interpret the difference. But there is a prior question that almost no one asks, and it can dissolve the finding entirely: does the instrument measure the same thing, on the same scale, in both groups? If it does not, the gap you are admiring may live in the measure, not in the people.

This is the problem of measurement invariance. When you compare scores across groups, you are quietly assuming the items behave the same way for everyone. The assumption can fail. When it does, you have what psychometricians call differential item functioning: two people with exactly the same true level of the trait respond differently to an item depending on which group they belong to.

An example makes it concrete. Suppose a depression scale includes the item ‘I cry easily.’ If, at the same underlying level of depression, women are more likely to endorse that item than men because of social norms about crying rather than because of depression, then the item adds to women’s scores for a reason that has nothing to do with the construct. Compare raw totals and you will find a gender difference in depression that is partly just a difference in how one item behaves. The ruler is bent differently for the two groups, and the bend is invisible in the totals.

This quietly threatens a huge share of routine comparisons. Across demographic groups, where items may carry different connotations. Across translations, where the Spanish and English versions of a questionnaire may not be equivalent no matter how careful the translation. Across countries, where a response scale is read differently. And across time, the most treacherous case: if a program changes how people interpret the questions, their internal yardstick moves, and that shift can masquerade as real change in the outcome.

The discipline is to test for invariance before you compare, not after. The approach builds up in steps. Configural invariance asks whether the same basic structure holds in each group, the same items tapping the same factors. Metric invariance adds the requirement that items relate to the construct with equal strength, which lets you compare relationships across groups. Scalar invariance adds equal intercepts, the level you actually need before comparing group means honestly. Reach it, and a difference in scores can be read as a difference in the trait; fall short, and the comparison is contaminated. Multi-group factor analysis and item response models make all of this testable rather than assumed.

None of this means a comparison is doomed the moment one item misbehaves. Partial invariance, where most items are equivalent and a few are not, is often enough to support a careful comparison once the offending items are handled. And here is the part worth holding onto: failing an invariance test is not a failure of your study. It is a finding. It tells you the construct is understood or expressed differently across the groups you care about, which is often substantively interesting in its own right. The real error is never running the test, and reporting the raw gap as though it were obviously real.

For those of us in federal evaluation, this lands close to home. Our work is saturated with exactly the comparisons that invariance governs: across states, across demographic subgroups, across program sites, across languages, and before and after an intervention. Equity analyses and subgroup breakouts depend entirely on the instrument meaning the same thing for every group being compared. Report a disparity or a pre-post gain without checking that, and you may be reporting a property of your questionnaire rather than a fact about the world.

So here is my question. Before you compare scores across groups or across time, do you check that the instrument means the same thing in each, or do you treat the numbers as automatically comparable?