Stephen T’s Blog Spot

The standard picture of evaluation assumes a program that stays put. You specify the model, define the goals, let it run for a while, and then judge whether it hit them. When the program is stable and well understood, that works well, and most of the methods this series has covered are built for it. But a great deal of real work is not like that. Genuinely new initiatives are still finding their shape. Complex efforts operate inside systems that keep shifting under them. Pilots are meant to learn, not to prove. Hold any of these against a fixed plan and you end up measuring a moving target with a frozen ruler.

The mismatch is deeper than it first appears. The two familiar modes of evaluation both assume there is a settled model to work with. Summative evaluation asks whether the program met predefined outcomes; formative evaluation helps improve the program toward its intended design, on the way to a later summative judgment. Both take the design as given. But for a true innovation, the design is exactly what is not yet settled, because building the model is the work. Forcing predefined outcomes and fidelity to a frozen plan onto such a program does more than miss the point: it punishes the adaptation the program should be doing, and it can lock in a design before anyone knows what actually works.

Michael Quinn Patton named the alternative developmental evaluation. Instead of judging a program against a fixed standard at the end, the evaluator works alongside the effort and supplies rigorous, timely evidence that feeds continuous adaptation. The purpose is development, not a verdict, and the guiding question changes from did it meet the targets to what is emerging, what is that telling us, and how should the program change in response. Crucially, this is not the same as formative evaluation, a distinction Patton stresses. Formative work improves a program toward a known model; developmental evaluation supports a program that has no settled model yet, and may never hold still long enough to acquire one.

It is worth being clear about what this is not, because the approach is easy to abuse. It is not a license to drop rigor, skip measurement, or make things up as you go. The evidence stays disciplined and documented; what is allowed to evolve is the goals and measures themselves, as understanding grows. Nor does it retire summative evaluation for good. Once an innovation stabilizes into a model that can be named and repeated, the standard questions return in full force: does this work, and is it worth it. Developmental evaluation is for the developmental phase, not a permanent posture.

The hard part is the tradeoff it forces on independence. To feed real-time evidence into a program, the evaluator has to sit close to it, embedded in the team rather than standing outside as a detached judge. That closeness is what makes the approach useful, and it is also its central risk: an evaluator this involved can slide from candid feedback into cheerleading, losing the honesty that makes the evidence worth having. Staying useful and staying truthful at once is the real skill. And the approach fits complexity and innovation specifically; using it where a stable, well-understood program already exists would just be avoiding accountability under a fashionable name.

For those of us working with federal programs, this matters because agencies increasingly fund innovation, pilots, and complex cross-system initiatives, and then commission evaluations designed for stable programs. That mismatch sets the evaluation up to measure the wrong thing and sets the program up to look like a failure precisely for adapting. Choosing the evaluation approach to fit the program’s actual nature, settled or still developing, is a design decision that should come before any choice of method. Sometimes the most rigorous option is the one that can keep pace with a program still finding its form.

So here is my question. When you evaluate something still being invented, do you hold it to a fixed plan it has outgrown, or match the evaluation to a program that is still taking shape?