Stephen T’s Blog Spot

A single number that ranks things carries enormous rhetorical power. States ordered by vulnerability, hospitals by quality, countries by competitiveness, programs by performance: the ranking looks like a measurement, objective and settled. But a composite index is not a measurement in the way a thermometer reading is. It is a construction, assembled from many separate pieces through a chain of choices, and those choices, as much as the underlying reality, decide who ends up on top.

Building an index means making at least four consequential decisions, each defensible and each arbitrary at the margin. First, which indicators to include, and which to leave out; the index measures only what you chose to put in it, so an omitted dimension simply vanishes from the ranking. Second, how to normalize them, since they arrive in different units and must be made comparable, and rescaling by z-scores, by a minimum-to-maximum stretch, or by ranks can give different results, especially with outliers. Third, how to weight them; equal weights feel neutral but are a strong assumption, and any set of weights is a statement about what matters more, usually chosen rather than derived. Fourth, how to aggregate; adding the pieces lets a high score compensate for a low one, while multiplying does not, so a weak spot cannot be bought back, and that choice alone can change the order.

Because every one of these choices moves the ranking, the order is partly an artifact of the recipe. Two competent analysts, each making entirely defensible choices, can produce different rankings from the very same data. Formal studies of index construction show exactly this: as you vary the weights and methods within reasonable bounds, a unit’s rank can swing, sometimes dramatically. Andrea Saltelli, one of the field’s authorities, warned that an index built without a sensitivity analysis can be made to tell almost any story. A rank presented as a fact is often a choice presented as a fact.

The real hazard is what this does to accountability. An index launders subjective judgments into an objective-looking number. The weighting decision, the most value-laden step of all, disappears into a formula, and the result emerges wearing the authority of arithmetic. People then treat the order as discovered rather than built, and they make real decisions, funding, targeting, oversight, on differences between adjacent ranks that fall well within the noise of the method. Position fourteen beats position fifteen, and a resource follows, when a slightly different but equally reasonable recipe would have swapped them.

None of this means indices are useless; it means they should be treated as models, not measurements. Make every choice explicit: what went in, how it was scaled, weighted, and combined, and why. Then run a sensitivity analysis, varying those choices to see how stable each position is, and report the instability instead of hiding it. Distinguish the gaps that survive reasonable variation from the ranks that reshuffle at a touch. And resist false precision: an index may be serviceable for sorting units into broad tiers, high, medium, and low, while being close to meaningless for ranking one position against the next.

For those of us working with federal programs, this is not abstract. Vulnerability indices, deprivation measures, risk scores, and performance rankings are used to direct money and to hold programs to account. When a funding formula or an oversight decision rests on an index rank, the construction choices behind that rank are not technical footnotes. They are policy choices in disguise, and they deserve the same scrutiny we would give any other policy choice.

So here is my question for the group. When you use a ranking, do you ask how it was built and how much the order would move under different reasonable choices, or do you treat the number as the thing itself?