Stephen T’s Blog Spot

You survey 1,500 people, and the sample size feels like a source of strength. Big N, tight confidence intervals, precise estimates. But suppose those 1,500 were not drawn independently from the population. They were reached by first selecting 60 neighborhoods and then interviewing 25 people in each. That single fact can quietly cut the real precision of your survey by more than half, and if you do not account for it, every standard error you report will be too small.

The reason is that people in the same cluster resemble each other. Households in a neighborhood share income levels, local conditions, and exposure to the same services. Students in a school share teachers and a common environment. Patients in a clinic share a provider and a protocol. Because the members of a cluster are more alike than two people picked at random from the whole population, each additional person you interview within a cluster tells you less that is genuinely new. You are, in part, hearing the same thing again.

Statisticians measure this similarity with the intraclass correlation, the share of the total variation that lies between clusters rather than within them. When that correlation is zero, clustering costs you nothing and your sample behaves like an independent one. When it is above zero, which it almost always is, the information in your data is less than the row count suggests. The tool that captures the damage is the design effect, and its logic is simple: the larger your clusters and the more alike people are within them, the more precision you lose compared to a truly random sample.

The numbers are more sobering than most people expect, because even weak within-cluster similarity adds up over large clusters. Take that survey of 1,500 people in 60 clusters of 25, and suppose the intraclass correlation is a modest 5 percent. The design effect works out to about 2.2, which means your effective sample size, the number of independent observations your data is really worth, is closer to 680 than to 1,500. You paid for 1,500 interviews and bought the precision of fewer than 700.

Now the practical danger comes into focus. If you analyze clustered data as though every observation were independent, you are dividing by the wrong sample size. Your standard errors come out too small, your confidence intervals too narrow, and your p-values too impressive. You will report precision you did not earn, and you will call differences significant that a proper analysis would leave in doubt. This is the same false-certainty problem this series has kept circling, arriving now from the sampling side rather than the significance side.

The fix is not to avoid clustering, which is often unavoidable and sometimes the only affordable way to collect data. The fix is to analyze the data the way it was collected. Design-based survey methods, cluster-robust standard errors, and multilevel models all exist to give clustered data honest uncertainty. And at the planning stage the design effect runs the other way: if you know you will cluster, you inflate your target sample size in advance to buy back the precision you are about to lose.

For those of us working with federal survey data and multisite program data, this is not a corner case. It is the normal structure of the work. People are sampled within schools, counties, facilities, and program sites, and the data arrive looking flat, one row per person, with the clustering invisible unless you know to look. Reporting a confident national estimate from clustered data analyzed as a simple random sample is one of the most common ways a rigorous-looking study overstates what it knows.

So here is my question. When you report the precision of an estimate from clustered data, do you count every row as an independent piece of information, or do you ask how many independent observations you truly have?