Stephen T’s Blog Spot

We pour enormous effort into getting the methods right: the design, the analysis, the careful caveats. Then the report is delivered, politely thanked, and set on a shelf, where nothing happens to it. In evaluation, the most common failure is not a flawed method. It is irrelevance. A technically flawless study that changes no decision has failed at the one thing it was for. Michael Quinn Patton built an entire approach around that problem, and called it utilization-focused evaluation.

Its premise fits in three words: intended use by intended users. That sounds too obvious to state, and it is routinely ignored. Most evaluations are aimed at an audience, which is an amorphous, anonymous entity, which is to say no one in particular. A report for everyone tends to be used by no one. Utilization-focused evaluation throws out the audience and replaces it with specific, named primary intended users and the specific decisions they actually have to make.

This rests on what Patton called the personal factor, the part worth remembering. Across his research the pattern repeated: evaluations got used when a real, identifiable person or small group personally cared about the findings, and went unused when that person was missing. The lesson is concrete. The program is not a user. The program director who will brief the board in March on whether to renew the contract is a user. That gap is the gap between a report that moves something and a report that gathers dust.

So in this approach you identify those people first, engage them early, and let their intended use drive every later choice: the questions you ask, the methods you pick, the timeline, even how you present results. Use, Patton argued, flows from clarity of purpose. His guiding question to a client is bracingly practical: what do you not know that, if you did know it, would change what you do? If a finding would not change any decision, it probably does not deserve the evaluation’s scarce attention.

There are two bonuses. Involving users early builds ownership, and people act on what they feel ownership of. And there is what Patton named process use: simply taking part in a serious evaluation changes how people think about their program, sometimes before a single finding arrives. Methods stay flexible throughout, quantitative, qualitative, or mixed, chosen for what will be useful rather than for disciplinary habit.

Now the honest tension. Getting close to the people who will use your work raises an obvious danger: that the evaluation drifts toward telling them what they want to hear, trading independence for relevance. Utilization-focused evaluation does not license that. Findings still have to be credible, or all you have achieved is the enthusiastic use of a flattering fiction. The skill is holding both at once, close enough to be used and independent enough to be trusted. Relevance without credibility is spin; credibility without relevance is the report on the shelf.

So before you design the next study, ask who specifically will act on it, and what decision they face. Then design backward from that, while guarding the independence that makes the findings worth acting on in the first place.

So here is my question: Before you start, do you name the actual person who will use the findings and the decision they need to make, and how do you stay useful to them without becoming captured by them?