Stephen T’s Blog Spot

You are asked the question every program eventually faces: what difference did it make? But there is no control group. You could not randomize. A dozen other things were happening at the same time, the economy, other programs, the simple passage of time. The counterfactual methods I have written about, matching, regression discontinuity, difference-in-differences, all need a comparison you simply do not have. And yet giving up on the causal question is not an option, because the people who funded the work still need to know whether it worked. This is the situation contribution analysis was built for.

Start with a shift in the question. Most rigorous causal methods chase attribution: how much of the outcome did the program cause, measured against a world where it never happened? Contribution analysis, developed by the evaluator John Mayne, asks something humbler and often more honest: did the program make a meaningful difference, as one cause among several? You are not claiming sole credit, and you are not producing a precise percentage. You are building a credible case that the program mattered, alongside everything else that was going on.

The backbone of that case is the theory of change from the last post. Mayne’s logic runs roughly like this. You can reasonably conclude a program contributed when four things hold. Its theory of change is plausible. The activities were actually carried out, not just planned. The chain of expected results in between is confirmed by evidence, not assumed. And the other plausible influences on the outcome have been examined and either ruled out or given their share of the credit. Meet those conditions and you have something defensible. The product is not an effect size. It is a reasoned, evidence-backed argument, what Mayne calls a contribution story.

Now the part that separates the method from a press release. The whole exercise lives or dies on that fourth condition, the alternative explanations. Anyone can assemble the evidence that flatters their program and arrange it into a tidy narrative. That is not analysis; it is marketing with footnotes. What makes contribution analysis rigorous is the obligation to go hunting for the rival explanations and to confront them head on. Maybe the outcome improved because the broader economy did. Maybe a second initiative was running in parallel and deserves the credit. Maybe the people served would have improved on their own. A contribution story that never names a single rival it had to argue against is not a finding. It is a testimonial.

Be honest about what you get and do not get. Contribution analysis yields a plausibility judgment and a stated level of confidence, not the clean number a randomized experiment delivers. Because it leans on reasoning rather than a counterfactual, it is only as strong as the evidence you bring and the seriousness of your search for alternatives. Many evaluators now pair it with process tracing, which sets explicit tests for how strong a piece of evidence needs to be before it counts. None of that turns it into an experiment. It is what you do responsibly when an experiment was never on the table.

And in real program work, it usually is not. So the discipline is this. When you cannot randomize, do not retreat to ‘we cannot say,’ and do not leap to ‘it obviously worked.’ Build the theory of change, gather evidence on every link in it, and then spend most of your energy on the explanations you would prefer were not true. The credibility of the whole claim rests there.

So here is my question: When you cannot build a counterfactual, how do you decide a program genuinely contributed, and how hard do you actually look for the alternative explanations that would undercut your story?