Stephen T’s Blog Spot

There is an old joke about a man searching for his lost keys under a streetlight. A passerby asks where he dropped them. “Over in the park,” he says, “but the light is better here.” We laugh, and then we go build our performance dashboards the same way.

This is the streetlight effect, and its more formal name is the McNamara fallacy, after Robert McNamara, who as Secretary of Defense leaned heavily on quantifiable measures of progress. The social scientist Daniel Yankelovich described it as a four-step slide. First, measure what can easily be measured. That is fine. Second, disregard what cannot easily be measured, or assign it an arbitrary number. That is misleading. Third, assume that what cannot be measured easily is not important. That, he said, is blindness. Fourth, conclude that what cannot be measured does not exist at all. That, he said, is suicide.

Nothing makes the trap more vivid than the current scramble to measure the productivity payoff of AI. A 2026 industry report covered by MIT Sloan Management Review found that nearly nine in ten engineering leaders believe AI tools improved their developers’ productivity, while much of the promised gain shows up in no metric at all. The new work AI creates, the reviewing, the validating, the cognitive load of checking a machine’s output, lands precisely in the dark, outside the dashboard.

It gets stranger. In a controlled study, experienced developers given AI tools actually took longer to finish their tasks, yet afterward they believed the tools had sped them up. The thing that was easy to feel, a sense of speed, pointed the opposite way from the thing that was hard to measure, the actual time. If even the people doing the work misjudge it, imagine what a tidy chart of commits and lines of code is really telling you.

That is the heart of the problem with knowledge work. The easy numbers, tickets closed, hours logged, lines written, measure motion, not value. They are real and not useless, but they are the light near the lamppost. The actual product, good judgment, a problem prevented, a hard call made well, tends to sit in the dark where counting is awkward. And the moment we decide the dark contains nothing worth knowing, we have taken Yankelovich’s fatal fourth step.

The answer is not to throw away the numbers. Quantitative metrics are fast, comparable, and honest about what they capture. The answer is to stop asking them to carry meaning they cannot hold, and to pair them with evidence that reaches into the dark. This is the case for mixed methods, the discipline I keep coming back to. Put the dashboard next to interviews, observation, open-ended questions, and expert judgment. Let the numbers tell you what changed and how much; let the qualitative work tell you what it meant and why. When a metric jumps, the right next question is often not statistical but human: go ask the people what actually happened.

The two kinds of evidence also check each other. Qualitative insight keeps you from worshiping a number that measures the wrong thing, and quantitative measures keep your stories honest and at scale. Neither is the senior partner. A serious evaluation needs both, precisely because the most important effects are so often the ones that do not fit neatly in a cell of a spreadsheet.

So here is my question: In your work, what is the most important thing you measure that does not fit in a dashboard, and how do you keep it from being treated as if it does not exist?