Stephen T’s Blog Spot

Here is a scene familiar to anyone who has done this work. A program has run for two years. The funder wants an evaluation, a contract is awarded, and a few months in the team hits a wall. Nobody quite agrees on what the program was supposed to achieve. The goals are vague. The data needed to measure outcomes was never collected. And it is unclear who will use the findings, or for what decision. The evaluation was in trouble before it began, not because the methods were weak, but because the program was never ready to be evaluated.

There is a step designed to catch exactly this, and it is decades old. Joseph Wholey and his colleagues at the Urban Institute developed evaluability assessment in the 1970s, in response to a wave of government program evaluations that arrived late and delivered little. The premise is simple. Before committing to a full evaluation, you check whether the program can be meaningfully evaluated at all.

An evaluability assessment asks a few hard questions up front. Is the program clearly defined, and do the people running it agree on its goals, its audience, and its activities? Is there a plausible theory of change, a credible account of how the activities produce the outcomes, rather than just a hope? Do the data needed to answer the question exist, or could they be gathered at reasonable cost? And, most important, is there a genuine audience for the findings, someone who intends to act on them? When the answer to several is no, a full evaluation is not yet worth funding.

The work itself is modest. It leans on document review, program plans, logic models, performance data, and on structured conversations with the people who matter: managers, funders, frontline staff, and intended beneficiaries. The goal is not to grade anyone. It is to surface, early and cheaply, the gaps that would otherwise sink the real evaluation months later.

So an evaluability assessment does not always say go. Often it says not yet, and recommends fixing the program’s logic, defining measurable objectives, or building the data collection before any outcome study begins. Wholey framed it as the first and cheapest purchase in a sequence: before you buy the expensive impact evaluation, you buy a small look at whether that evaluation could succeed. Michael Scriven put it more bluntly, likening evaluability to the basic serviceability you expect when buying a new car.

Here is the part people miss. Even when no evaluation follows, an evaluability assessment usually leaves the program better than it found it. Forcing stakeholders to articulate goals, draw out the theory of change, and name the intended use sharpens the program itself. It connects to two ideas from earlier in this series: the theory of change as a falsifiable account of how change is supposed to happen, and the principle that an evaluation exists to be used by specific people for specific decisions. Evaluability assessment is where both get checked before the meter starts running.

A caution, to keep it honest. This is not a way to dodge difficult evaluations or to stall forever. Used cynically, the claim that a program is not evaluable becomes a convenient excuse to avoid accountability. The discipline is to use it as Wholey intended: a fast diagnostic that either clears the path for a credible evaluation or tells you, plainly, what to fix first.

So before your next evaluation goes to contract, spend a little to ask whether the program is ready: clear goals, a real theory of change, available data, and a user waiting for the answer. It is far cheaper to learn a program is not evaluable in month one than in month nine.

So here is my question: Have you ever been brought in to evaluate a program that was not ready to be evaluated, and what would have changed if someone had asked the question at the start?