Stephen T’s Blog Spot

Every causal claim from observational data rests on an assumption that cannot be checked. When you estimate the effect of a program, a treatment, or a policy from data you did not randomize, you are assuming that you have measured and adjusted for every important confounder. There is no test for this. The variable that ruins your estimate is, by definition, the one you did not measure. So the honest question is not whether unmeasured confounding exists. It almost certainly does. The question is how much of it your conclusion could withstand.

This is what sensitivity analysis is for, and it is the discipline this series keeps circling back to. Methods like propensity-score matching, regression adjustment, and difference-in-differences all do their work on the confounders you can see. None of them touches the ones you cannot. Sensitivity analysis fills that gap, not by removing the hidden bias, but by asking a sharp and answerable question: how strong would an unmeasured confounder have to be to overturn this result?

The most accessible tool for this is the E-value, introduced by Tyler VanderWeele and Peng Ding in 2017. It answers that question with a single number. The E-value is the minimum strength of association, expressed as a risk ratio, that an unmeasured confounder would need to have with both the treatment and the outcome, above and beyond everything you already adjusted for, in order to fully explain away your finding. In plain terms: how strong would the thing you missed have to be to erase your effect?

The interpretation is direct. A large E-value means it would take a powerful hidden confounder to undo your result, so the finding is relatively robust. A small E-value means a fairly weak confounder, the kind that could plausibly be lurking, would be enough to reduce your effect to nothing. Suppose an analysis reports an E-value of 1.3. That says a confounder associated with both treatment and outcome by only about a 1.3-fold risk ratio each would suffice to explain the whole thing away. Confounders that weak are everywhere, so the result should not impress you. An E-value of 5, by contrast, demands a hidden factor stronger than most confounders anyone can name.

You can compute the E-value not just for the point estimate but for the limit of the confidence interval nearest the null. That version asks how much confounding it would take not to erase the effect entirely, but merely to make it statistically indistinguishable from zero. It is usually the more honest number to report, because it speaks to the boundary of the claim rather than its center.

What makes this a discipline rather than a calculation is the next step: comparing the E-value to what you know about the setting. A number alone means little. The question is whether a confounder of the required strength is plausible given the covariates you already controlled. If you have adjusted for the obvious drivers and the E-value still demands an implausibly strong hidden factor, your causal claim has earned some confidence. If a mundane unmeasured variable could clear the bar, temper the conclusion, whatever the p-value says. That comparison is a judgment informed by subject knowledge, not something the number decides for you.

For those of us who evaluate programs on observational data, which is most of us most of the time, this belongs in the standard toolkit. A confidence interval tells you about sampling noise. It says nothing about the larger threat, bias from what you could not measure. Reporting an effect without a sensitivity analysis shows only the risk from randomness while staying silent about the risk from confounding. The mature move is to state, in one defensible number, how strong the missing piece would have to be, and then argue honestly about whether it could be that strong.

So here is my question. When you present a causal estimate from observational data, do you quantify how much unmeasured confounding it would take to overturn it, or do you let the confidence interval stand in for a robustness it was never designed to measure?