Stephen T’s Blog Spot

A blog aimed at issues only data scientists, data analysts, statisticians, evaluators, and researchers care about.

recent posts

about

I spend my time thinking about how the government can use research and evaluation methods to make evidence informed decisions and policy. It can be a struggle! That is why I need your thoughts. Join me, have a read, and make a comment. Its how we add to the evidence base,

  • Looking Where the Light Is Good: The Streetlight effect and the McNamara Fallacy

    There is an old joke about a man searching for his lost keys under a streetlight. A passerby asks where he dropped them. “Over in the park,” he says, “but the light is better here.” We laugh, and then we go build our performance dashboards the same way.

    This is the streetlight effect, and its more formal name is the McNamara fallacy, after Robert McNamara, who as Secretary of Defense leaned heavily on quantifiable measures of progress. The social scientist Daniel Yankelovich described it as a four-step slide. First, measure what can easily be measured. That is fine. Second, disregard what cannot easily be measured, or assign it an arbitrary number. That is misleading. Third, assume that what cannot be measured easily is not important. That, he said, is blindness. Fourth, conclude that what cannot be measured does not exist at all. That, he said, is suicide.

    Nothing makes the trap more vivid than the current scramble to measure the productivity payoff of AI. A 2026 industry report covered by MIT Sloan Management Review found that nearly nine in ten engineering leaders believe AI tools improved their developers’ productivity, while much of the promised gain shows up in no metric at all. The new work AI creates, the reviewing, the validating, the cognitive load of checking a machine’s output, lands precisely in the dark, outside the dashboard.

    It gets stranger. In a controlled study, experienced developers given AI tools actually took longer to finish their tasks, yet afterward they believed the tools had sped them up. The thing that was easy to feel, a sense of speed, pointed the opposite way from the thing that was hard to measure, the actual time. If even the people doing the work misjudge it, imagine what a tidy chart of commits and lines of code is really telling you.

    That is the heart of the problem with knowledge work. The easy numbers, tickets closed, hours logged, lines written, measure motion, not value. They are real and not useless, but they are the light near the lamppost. The actual product, good judgment, a problem prevented, a hard call made well, tends to sit in the dark where counting is awkward. And the moment we decide the dark contains nothing worth knowing, we have taken Yankelovich’s fatal fourth step.

    The answer is not to throw away the numbers. Quantitative metrics are fast, comparable, and honest about what they capture. The answer is to stop asking them to carry meaning they cannot hold, and to pair them with evidence that reaches into the dark. This is the case for mixed methods, the discipline I keep coming back to. Put the dashboard next to interviews, observation, open-ended questions, and expert judgment. Let the numbers tell you what changed and how much; let the qualitative work tell you what it meant and why. When a metric jumps, the right next question is often not statistical but human: go ask the people what actually happened.

    The two kinds of evidence also check each other. Qualitative insight keeps you from worshiping a number that measures the wrong thing, and quantitative measures keep your stories honest and at scale. Neither is the senior partner. A serious evaluation needs both, precisely because the most important effects are so often the ones that do not fit neatly in a cell of a spreadsheet.

    So here is my question: In your work, what is the most important thing you measure that does not fit in a dashboard, and how do you keep it from being treated as if it does not exist?

  • From Two Points to Two Trends: Controlled Interrupted Time Series

    In another post I described difference-in-differences, the design that estimates a policy’s effect by comparing the change in a treated group to the change in an untreated one. Today I want to build on it, because there is a more powerful relative of that design, one I have leaned on often: the controlled interrupted time series, or CITS. It is best understood as difference-in-differences with the trends filled in.

    Recall the limitation hiding inside basic difference-in-differences. In its simplest form it uses just two snapshots, before and after, for each group, comparing how much the treated group’s average moved against the comparison group’s. That works, but it is blind to everything between and around those two points. It cannot see whether the groups were already on different trajectories before the policy, which is exactly where the parallel trends assumption quietly fails.

    An interrupted time series fixes the blindness by using many measurements over time rather than two. You track the outcome across numerous points before the intervention, establish the pre-existing trend, and then test whether the series breaks at the moment of the intervention, in its level, its slope, or both. The interruption is the policy. The projected continuation of the old trend is the counterfactual.

    A single interrupted time series still has a weakness. Something else might have happened at the same moment as your intervention, a recession, a seasonal swing, a national change, and you could mistake its effect for your program’s. This is where the control comes in. A controlled interrupted time series runs two trend analyses side by side: one for the treated group and one for a comparison group that was not exposed. If the treated series breaks from its own trend at the intervention and the comparison series does not, you have strong evidence the break was caused by the policy, not by something in the air that month.

    Here is the relationship that matters. Basic difference-in-differences asks whether the treated group’s level jumped more than the comparison group’s between two points. CITS asks whether the treated group’s entire trajectory, its level and its slope, broke from its established trend by more than the comparison group’s did, using the full time series. In that sense, difference-in-differences is a simplified, two-period special case of the richer design. CITS earns its added power by relaxing the parallel trends assumption: instead of assuming the groups would have moved in parallel, it models each group’s actual trend and asks whether the treated one deviated.

    That power is not free. CITS has heavier data requirements: you generally need several measurements before the intervention, at least four and ideally many more, to estimate the baseline trend, plus a good run afterward. Time series data also break a standard statistical assumption, because consecutive observations are correlated, so the analysis, usually segmented regression, must be adjusted for that autocorrelation or the uncertainty will be understated. And you still need a genuinely unaffected comparison series.

    For evaluators, the practical appeal is real. When a program launches at a known date and you have a decent history of outcome data for both an affected and an unaffected group, CITS lets you separate a real, sustained change from noise, pre-existing momentum, and coincident events, more convincingly than a two-point comparison can. It is one of the strongest quasi-experimental designs available when randomizing is off the table.

    So here is my question: When you have a clear intervention date and a run of historical data, do you reach for difference-in-differences or step up to a controlled interrupted time series, and what tips the decision for you?

  • When One Place Changes and Another Does not: Difference-in-Differences

    In a separate blog, I wrote about synthetic controls, a clever way to build a comparison group when you do not have one. Today, its simpler and far more common cousin, a workhorse of policy evaluation: difference-in-differences.

    Start with the problem it solves. A policy changes in one place and not another. One state raises a tax, passes a law, or launches a program; a neighbor does not. You want to know the effect. The naive approach is to look at the treated place before and after. But a simple before-and-after cannot work, because the world does not hold still. The economy shifts, the seasons turn, new trends arrive. Any change you see is a tangle of the policy and everything else that happened in the meantime.

    Difference-in-differences cuts through the tangle with a second difference. You track the treated group before and after, and you track an untreated comparison group over the same period. The comparison group is also exposed to the economy, the seasons, and the trends, just not to the policy. So whatever happened to the comparison group is your estimate of what would have happened to the treated group anyway. Subtract that out, and what remains is the effect of the policy. The first difference is before versus after. The second difference is treated versus comparison. Hence the name.

    The famous example is a 1994 study by David Card and Alan Krueger. In 1992, New Jersey raised its minimum wage while neighboring Pennsylvania did not. The researchers surveyed fast-food restaurants in both states before and after. Employment in Pennsylvania drifted down over those months; had New Jersey simply followed, its employment should have drifted down too. Instead it held roughly steady. The difference between the two differences was the estimated effect, and it ran against the textbook prediction that a higher minimum wage cuts jobs. The result was influential and hotly debated for years, a lesson in itself about how a single study lands.

    The whole method rests on one assumption, and it is worth saying plainly because it is so easy to skip past. It is called parallel trends: the idea that, absent the policy, the treated and comparison groups would have moved in parallel. You can never prove this, because you cannot observe the world where the policy did not happen. What you can do is check whether the two groups moved together before the intervention. If they tracked each other for years beforehand, the assumption is more believable. If they were already diverging, difference-in-differences will hand you a confident, wrong answer.

    And there are traps even when the trends look parallel. The comparison group must be genuinely unaffected; if New Jersey’s hike pulled workers across the border, Pennsylvania is contaminated. Other changes sneak in too: New Jersey also cut its sales tax that same year, so part of any effect could belong to the tax, not the wage. A clean difference-in-differences needs a comparison that is truly comparable, truly untouched, and not subject to some other shock at the same moment.

    For those of us evaluating policies and programs, this is one of the most useful tools there is, precisely because the real world keeps handing us situations where one group changed and a similar one did not. But it is only as good as its comparison and its assumption. The discipline is to show the pre-period trends, argue honestly for parallel trends rather than assert it, and hunt for the other things that changed at the same time.

    So here is my question: When you have used a difference-in-differences design, how did you defend the parallel trends assumption, and what convinced you the comparison group was really a fair stand-in?

  • Triangulation Is Not a Validity Stamp

    Last time I argued that mixed methods is really about integration. There is one word that often stands in for that integration and quietly does a great deal of unearned work: triangulation. People write ‘we triangulated’ as though it settles the question of whether a finding can be trusted. By itself, it does not.

    Start with what the word is doing. Triangulation is only one of several reasons to combine methods. Studying fifty-seven mixed-method evaluations in 1989, Jennifer Greene and colleagues found that researchers mix methods for at least five distinct purposes, and triangulation is just one. The others are complementarity, using one method to elaborate another; development, letting one method’s results build the other, as when interviews shape a survey; initiation, deliberately seeking contradiction to provoke insight; and expansion, using different methods for different parts of the inquiry. Many studies that claim triangulation are actually doing one of these four. Naming the purpose honestly tells you what convergence would even mean.

    Now the condition almost everyone skips. Triangulation, in its strict sense, means using more than one method to study the same phenomenon and looking for convergence, in order to cancel out the bias of any single method. The logic depends entirely on independence. It only works if the methods have different weaknesses, so that their errors do not line up. Two methods that share a bias do not corroborate each other; they reinforce the same mistake. If your survey and your interviews both depend on people remembering accurately and answering honestly, their agreement is comforting and may still be wrong. Convergence from methods that share a blind spot is not validation. It is an echo.

    There is also a quieter confusion at work, and it is the most common one. When you use a survey and a set of interviews to study the same thing, you are often not measuring the same thing at all. The survey captures how often; the interviews capture why, and how it feels. Those are different facets, and they will not converge, because they were never aimed at the same point. Expecting them to agree, and fretting when they do not, mistakes complementarity for triangulation. The two methods are handing you a fuller picture, not a second opinion on the same number.

    And when methods genuinely are aimed at the same target and still disagree, that is not a failure to be smoothed away. As I have said before, the disagreement is often the most informative thing you have. It can mean one method is biased, or that the phenomenon is more complicated than either method alone would suggest. Greene’s framework even gives the deliberate pursuit of that tension a name: initiation.

    So ‘we triangulated’ should not function as a verbal stamp of validity. The useful questions are sharper than that. Were the methods actually aimed at the same construct, or at different facets of it? Did they carry independent weaknesses, or the same blind spot? And if they diverged, what is that divergence telling you? Answer those, and triangulation becomes a real instrument for strengthening an inference. Skip them, and it is just a reassuring word in a methods section.

    For those of us in evaluation, triangulation is almost a reflex in a proposal. The discipline is to make it mean something. Say which methods are converging on which construct, explain why their weaknesses are independent, and state what you will do if they disagree. That is the line between triangulation as a method and triangulation as decoration.

    So here is my question: When you see triangulation claimed in a study, do you check whether the methods were truly independent and aimed at the same thing, or has the word become a stamp that gets applied whenever more than one method is in the room?

  • Can This Program Be Evaluated?

    Here is a scene familiar to anyone who has done this work. A program has run for two years. The funder wants an evaluation, a contract is awarded, and a few months in the team hits a wall. Nobody quite agrees on what the program was supposed to achieve. The goals are vague. The data needed to measure outcomes was never collected. And it is unclear who will use the findings, or for what decision. The evaluation was in trouble before it began, not because the methods were weak, but because the program was never ready to be evaluated.

    There is a step designed to catch exactly this, and it is decades old. Joseph Wholey and his colleagues at the Urban Institute developed evaluability assessment in the 1970s, in response to a wave of government program evaluations that arrived late and delivered little. The premise is simple. Before committing to a full evaluation, you check whether the program can be meaningfully evaluated at all.

    An evaluability assessment asks a few hard questions up front. Is the program clearly defined, and do the people running it agree on its goals, its audience, and its activities? Is there a plausible theory of change, a credible account of how the activities produce the outcomes, rather than just a hope? Do the data needed to answer the question exist, or could they be gathered at reasonable cost? And, most important, is there a genuine audience for the findings, someone who intends to act on them? When the answer to several is no, a full evaluation is not yet worth funding.

    The work itself is modest. It leans on document review, program plans, logic models, performance data, and on structured conversations with the people who matter: managers, funders, frontline staff, and intended beneficiaries. The goal is not to grade anyone. It is to surface, early and cheaply, the gaps that would otherwise sink the real evaluation months later.

    So an evaluability assessment does not always say go. Often it says not yet, and recommends fixing the program’s logic, defining measurable objectives, or building the data collection before any outcome study begins. Wholey framed it as the first and cheapest purchase in a sequence: before you buy the expensive impact evaluation, you buy a small look at whether that evaluation could succeed. Michael Scriven put it more bluntly, likening evaluability to the basic serviceability you expect when buying a new car.

    Here is the part people miss. Even when no evaluation follows, an evaluability assessment usually leaves the program better than it found it. Forcing stakeholders to articulate goals, draw out the theory of change, and name the intended use sharpens the program itself. It connects to two ideas from earlier in this series: the theory of change as a falsifiable account of how change is supposed to happen, and the principle that an evaluation exists to be used by specific people for specific decisions. Evaluability assessment is where both get checked before the meter starts running.

    A caution, to keep it honest. This is not a way to dodge difficult evaluations or to stall forever. Used cynically, the claim that a program is not evaluable becomes a convenient excuse to avoid accountability. The discipline is to use it as Wholey intended: a fast diagnostic that either clears the path for a credible evaluation or tells you, plainly, what to fix first.

    So before your next evaluation goes to contract, spend a little to ask whether the program is ready: clear goals, a real theory of change, available data, and a user waiting for the answer. It is far cheaper to learn a program is not evaluable in month one than in month nine.

    So here is my question: Have you ever been brought in to evaluate a program that was not ready to be evaluated, and what would have changed if someone had asked the question at the start?

  • Real-World Data Is Not Real-World Evidence

    One of the biggest shifts in how evidence gets made is the rise of real-world data and real-world evidence. It began at the FDA and is spreading across health agencies and into evidence-based policy more broadly. The appeal is obvious: instead of slow, narrow, expensive clinical trials, use the mountain of data the system already generates, electronic health records, insurance claims, disease registries, wearables, to learn what actually works in the world. It is a powerful idea, and it is also where nearly every methods problem in this series shows up at once.

    Start with a distinction the FDA is careful to make, and so should we. Real-world data is the data: information on health and care collected routinely, outside of any trial. Real-world evidence is the conclusion about whether something works, derived from analyzing that data. Those are not the same thing. Data is not evidence. The phrase real-world evidence is earned by what you do to the data, not granted by where the data came from. The entire question is the leap from one to the other.

    The first reason that leap is hard is fitness for purpose. Real-world data was generated for care, billing, or operations, never to answer your research question. A diagnosis code exists to justify a payment, not to record clinical truth. So the first thing to ask, and the first thing the FDA asks, is whether the data is complete, accurate, and reliable enough, and whether it even captures the right population, exposure, and outcome for this question. A dataset can be perfectly fit for one question and useless for the next. A huge but ill-fitting dataset is not an asset. It is a confident-looking trap.

    The second reason is deeper. Setting aside trials that build randomization into routine care, most real-world data is not randomized. Who received which treatment was decided by doctors and patients for reasons bound up with the outcome. So real-world evidence inherits every threat this series has worked through: selection bias, confounding, data that is missing in non-random ways, and gaps in who appears in the database at all. The FDA’s second question is precisely whether the study design can support a causal claim. And scale offers no rescue. A hundred million biased records give you an extremely precise wrong answer.

    The field’s best response is to stop treating the data as a shortcut around a trial and start using it to emulate one. Target trial emulation, now central to thinking at the FDA and its counterparts abroad, forces you to specify the randomized trial you would have run, eligibility, treatment, outcome, follow-up, and then build that design within the data. The discipline surfaces the biases instead of burying them. Pragmatic trials go further, embedding real randomization inside ordinary care, buying a trial’s internal validity while keeping real-world reach. This is how raw data becomes credible evidence, and even then it rests, honestly, on assumptions about unmeasured confounding that cannot be fully tested.

    None of this is a knock on real-world evidence. It answers questions trials cannot: long-term and rare outcomes, the full messy population that trials screen out, and situations where a trial would be impossible or unethical, all faster and at lower cost. Done well, it complements trials rather than replacing them. The point is narrower and sharper. Real-world and big describe the data. Only design and scrutiny turn it into evidence.

    So when someone hands you real-world evidence, ask the two questions in order. Was the data fit to answer this question, and was the design strong enough to carry a causal claim? And when you produce it, earn the word evidence with the design, then go looking for the confounder that would overturn you.

    So here is my question: As real-world evidence shows up in more decisions, how do you separate the genuinely well-designed studies from the ones trading on the size and the real-world label alone?

  • Write the Trial You Wish You Could Run

    The most famous cautionary tale in observational research goes like this. For years, study after study found that women on hormone replacement therapy had healthier hearts, and it was widely prescribed partly for that reason. Then a large randomized trial, the Women’s Health Initiative, found the opposite: the therapy raised cardiac risk. A great deal of practice had been built on a backward result. The reckoning that followed gave us the best discipline we have for analyzing observational data, and it is the repair to last time’s problem: target trial emulation.

    When researchers went back to reconcile the two findings, the explanation was less mysterious than expected. The gap was largely driven by design choices, especially when women began therapy relative to menopause and how long they were followed, rather than by some hidden flaw in the data. The data was not the main villain. The analysis was. And that is the encouraging part, because analysis is something we control.

    The deadliest observational errors are the ones you build in by accident. Immortal time bias is the classic: if you sort people into groups by something that happens later, like who eventually started a drug, you quietly credit that group with a window of time in which they could not possibly have had the outcome, because they had to survive long enough to start. Prevalent-user bias is studying the people already on a treatment, the survivors who tolerated it, rather than new starters. Misaligned time zero is starting the clock at a different moment for each group. None of these requires an unmeasured confounder. They are own-goals, and any one of them can reverse a result on its own.

    Target trial emulation, developed by Miguel Hernan and James Robins, is built to prevent exactly these. The rule is simple and strict: before you touch the data, write the protocol of the randomized trial you wish you could run. Spell out who is eligible, the treatment strategies you are comparing, when follow-up begins, the outcome, the follow-up window, and the precise effect you mean to estimate. Then emulate each piece in your data. Enroll the eligible, classify treatment at the start as a new-user design, start everyone’s clock together, and adjust for the baseline factors you measured. Writing the protocol is what does the work, because a real trial would never permit immortal time, prevalent users, or a clock that starts whenever convenient, so emulating one drags those errors into the light before they take hold.

    Be clear about what this does and does not buy you. Target trial emulation is not a randomizer. It cannot conjure away unmeasured confounding, the one problem only real random assignment solves. What it does is separate the two kinds of trouble. It eliminates the self-inflicted, design-induced biases outright, and leaves the genuine assumption, that you measured the things that matter, in the open where it can be stated, argued, and stress-tested with a sensitivity analysis. That clarity is the value. It turns ‘we ran some regressions on a database’ into ‘here is the trial we emulated, and here is the one assumption it rests on.’

    So the practical rule is almost embarrassingly simple. Any time you are about to compare groups in administrative or real-world data, write the target-trial protocol first. Define time zero, use a new-user design, name the estimand before you run anything. It is the cheapest, most effective safeguard against fooling yourself, and it is fast becoming what regulators and careful reviewers expect to see.

    So here is my question: When you analyze observational data, do you write down the trial you are emulating before you start, and has that discipline ever caught a self-inflicted bias before it reached your conclusions?

  • A Good Instrument Is Hard to Find

    Across several posts on causal inference, propensity-score matching, regression discontinuity, target trial emulation, I kept hitting the same wall. Each method could adjust for the confounders you had measured, but none could touch the ones you had not. Unmeasured confounding was the shared limit. There is one classic method built to climb that wall, with a clever trick. It is called instrumental variables.

    The trick is to borrow a piece of randomness from the world. An instrument is a variable that nudges people toward or away from the treatment but has no other route to the outcome. If you can find one, the part of the treatment driven by the instrument behaves as though it were randomized, and you can use that slice of variation to estimate a causal effect even while unmeasured confounders lurk. The classic example is quarter of birth. Because of school-entry and compulsory-attendance rules, children born at different times of year once ended up with slightly different amounts of schooling, for reasons unrelated to ability or family. Joshua Angrist and Alan Krueger used that accident of timing to estimate the effect of schooling on earnings. The draft lottery is another: random numbers pushed some men into military service, letting researchers estimate the effect of service on later earnings.

    The power comes entirely from the instrument being valid, and validity has three parts. First, relevance: the instrument has to actually move the treatment, and move it strongly. Second, the exclusion restriction: it must affect the outcome only through the treatment, with no side door. Third, exogeneity: it must be unrelated to the unmeasured confounders, as if randomly assigned. The first part you can check in the data. The other two, which carry the entire causal claim, you mostly cannot. They rest on an argument about how the world works, not on a statistical test.

    This is the uncomfortable part. A variable can be a strong predictor of the treatment and still be a terrible instrument, because it reaches the outcome through some other path. When that happens, instrumental variables does not reduce bias. It can amplify it, and hand you a precise, confident, wrong answer. A weak instrument is its own hazard: if it barely moves the treatment, the estimate turns unstable and biased, exactly the problem later critics found hiding in some famous quarter-of-birth results. The method is powerful and fragile at once, and a bad instrument is worse than none.

    Even a strong, valid instrument answers a narrower question than most readers assume. It estimates the effect only for the compliers, the people whose behavior the instrument actually changed. In the draft lottery, that means the men who served because they were drafted and would not have otherwise, not the volunteers, and not those who would have avoided service regardless. The result is a local average treatment effect. It need not generalize to everyone, and reporting it as the effect for everyone is a quiet overreach.

    So instrumental variables is the rare tool that genuinely targets unmeasured confounding, the wall the other designs could not climb. But it does not make the problem vanish. It relocates it. Instead of assuming you measured every confounder, you now must defend, by argument rather than data, that your instrument has no back door and was as good as random. Often that is just as hard. The honesty lies in stating which assumption your causal claim leans on, and admitting it cannot be proven.

    For those of us in evaluation, real instruments are rare and valuable: a lottery for an oversubscribed program, a policy cutoff, a randomized nudge to participate. When you have one, it can rescue a question that observational data alone could not answer. But resist the urge to christen any convenient variable an instrument. Ask the hard question out loud: could this thing affect the outcome through any path other than the program? If you cannot rule that out, you do not have an instrument. You have another confounder.

    So here is my question: Have you ever found a genuinely credible instrument in your own work, and did its exclusion restriction survive a skeptical second look?

  • Who Is This Evaluation Actually For?

    We pour enormous effort into getting the methods right: the design, the analysis, the careful caveats. Then the report is delivered, politely thanked, and set on a shelf, where nothing happens to it. In evaluation, the most common failure is not a flawed method. It is irrelevance. A technically flawless study that changes no decision has failed at the one thing it was for. Michael Quinn Patton built an entire approach around that problem, and called it utilization-focused evaluation.

    Its premise fits in three words: intended use by intended users. That sounds too obvious to state, and it is routinely ignored. Most evaluations are aimed at an audience, which is an amorphous, anonymous entity, which is to say no one in particular. A report for everyone tends to be used by no one. Utilization-focused evaluation throws out the audience and replaces it with specific, named primary intended users and the specific decisions they actually have to make.

    This rests on what Patton called the personal factor, the part worth remembering. Across his research the pattern repeated: evaluations got used when a real, identifiable person or small group personally cared about the findings, and went unused when that person was missing. The lesson is concrete. The program is not a user. The program director who will brief the board in March on whether to renew the contract is a user. That gap is the gap between a report that moves something and a report that gathers dust.

    So in this approach you identify those people first, engage them early, and let their intended use drive every later choice: the questions you ask, the methods you pick, the timeline, even how you present results. Use, Patton argued, flows from clarity of purpose. His guiding question to a client is bracingly practical: what do you not know that, if you did know it, would change what you do? If a finding would not change any decision, it probably does not deserve the evaluation’s scarce attention.

    There are two bonuses. Involving users early builds ownership, and people act on what they feel ownership of. And there is what Patton named process use: simply taking part in a serious evaluation changes how people think about their program, sometimes before a single finding arrives. Methods stay flexible throughout, quantitative, qualitative, or mixed, chosen for what will be useful rather than for disciplinary habit.

    Now the honest tension. Getting close to the people who will use your work raises an obvious danger: that the evaluation drifts toward telling them what they want to hear, trading independence for relevance. Utilization-focused evaluation does not license that. Findings still have to be credible, or all you have achieved is the enthusiastic use of a flattering fiction. The skill is holding both at once, close enough to be used and independent enough to be trusted. Relevance without credibility is spin; credibility without relevance is the report on the shelf.

    So before you design the next study, ask who specifically will act on it, and what decision they face. Then design backward from that, while guarding the independence that makes the findings worth acting on in the first place.

    So here is my question: Before you start, do you name the actual person who will use the findings and the decision they need to make, and how do you stay useful to them without becoming captured by them?

  • A Good Instrument Is Hard to Find

    Across several previous posts on causal inference, propensity-score matching, regression discontinuity, target trial emulation, I kept hitting the same wall. Each method could adjust for the confounders you had measured, but none could touch the ones you had not. Unmeasured confounding was the shared limit. There is one classic method built to climb that wall, with a clever trick. It is called instrumental variables.

    The trick is to borrow a piece of randomness from the world. An instrument is a variable that nudges people toward or away from the treatment but has no other route to the outcome. If you can find one, the part of the treatment driven by the instrument behaves as though it were randomized, and you can use that slice of variation to estimate a causal effect even while unmeasured confounders lurk. The classic example is quarter of birth. Because of school-entry and compulsory-attendance rules, children born at different times of year once ended up with slightly different amounts of schooling, for reasons unrelated to ability or family. Joshua Angrist and Alan Krueger used that accident of timing to estimate the effect of schooling on earnings. The draft lottery is another: random numbers pushed some men into military service, letting researchers estimate the effect of service on later earnings.

    The power comes entirely from the instrument being valid, and validity has three parts. First, relevance: the instrument has to actually move the treatment, and move it strongly. Second, the exclusion restriction: it must affect the outcome only through the treatment, with no side door. Third, exogeneity: it must be unrelated to the unmeasured confounders, as if randomly assigned. The first part you can check in the data. The other two, which carry the entire causal claim, you mostly cannot. They rest on an argument about how the world works, not on a statistical test.

    This is the uncomfortable part. A variable can be a strong predictor of the treatment and still be a terrible instrument, because it reaches the outcome through some other path. When that happens, instrumental variables does not reduce bias. It can amplify it, and hand you a precise, confident, wrong answer. A weak instrument is its own hazard: if it barely moves the treatment, the estimate turns unstable and biased, exactly the problem later critics found hiding in some famous quarter-of-birth results. The method is powerful and fragile at once, and a bad instrument is worse than none.

    Even a strong, valid instrument answers a narrower question than most readers assume. It estimates the effect only for the compliers, the people whose behavior the instrument actually changed. In the draft lottery, that means the men who served because they were drafted and would not have otherwise, not the volunteers, and not those who would have avoided service regardless. The result is a local average treatment effect. It need not generalize to everyone, and reporting it as the effect for everyone is a quiet overreach.

    So instrumental variables is the rare tool that genuinely targets unmeasured confounding, the wall the other designs could not climb. But it does not make the problem vanish. It relocates it. Instead of assuming you measured every confounder, you now must defend, by argument rather than data, that your instrument has no back door and was as good as random. Often that is just as hard. The honesty lies in stating which assumption your causal claim leans on, and admitting it cannot be proven.

    For those of us in evaluation, real instruments are rare and valuable: a lottery for an oversubscribed program, a policy cutoff, a randomized nudge to participate. When you have one, it can rescue a question that observational data alone could not answer. But resist the urge to christen any convenient variable an instrument. Ask the hard question out loud: could this thing affect the outcome through any path other than the program? If you cannot rule that out, you do not have an instrument. You have another confounder.

    So here is my question: Have you ever found a genuinely credible instrument in your own work, and did its exclusion restriction survive a skeptical second look?

  • Understanding Publication Bias and the File-Drawer Problem

    Understanding Publication Bias and the File-Drawer Problem

    We live in the age of evidence-based everything: evidence-based policy, evidence-based practice, evidence-based design. When someone says “the research shows,” it is meant to end the argument. But a quiet problem is buried in that phrase. The research you get to see is not a fair sample of the research that was actually done.

    The problem has a memorable name, coined by the psychologist Robert Rosenthal in 1979: the file drawer problem. Studies that find a strong, statistically significant effect tend to get written up, submitted, and published. Studies that find nothing, a null result, tend to get quietly filed away and forgotten. The finished research that reaches you has passed through a filter, and the filter favors exciting results over boring ones.

    How big is the filter? In a striking 2014 study in Science, researchers tracked a known population of social science experiments, so they knew which ones were run and what each found. Among studies with strong results, most were published. Among those that came up null, only about one in five made it into a journal. And the biggest reason was not that journals rejected the nulls. It was that the authors never even wrote them up. The dead ends went straight into the drawer.

    The consequence is visible in the literature itself. Across fields, the overwhelming majority of published studies, often well over ninety percent, report a statistically significant finding. That cannot be what real research looks like, because real research is full of dead ends and things that do not pan out. What we are seeing is not the truth about how often effects are real. We are seeing the truth about what gets published.

    This matters for anyone who relies on a body of evidence. Run a literature review or meta-analysis using only published studies, and you are averaging a sample stripped of its failures. The effect you calculate comes out too big, sometimes dramatically, because the studies that found nothing sit invisibly in a thousand drawers. It is a main reason impressive findings shrink or vanish when someone tries to replicate them.

    This is the companion to a problem I have written about before. P-hacking manufactures false positives inside one study; publication bias decides which finished studies you ever see. One distorts the result, the other the whole library. Together they make a field look more certain than it is.

    The fixes are mostly structural and gaining ground. Study registries record what researchers plan before they do it, revealing which studies were started and never reported. Registered reports, where a journal commits to publish based on the design before results are known, break the link between significance and publication. And in reviews, tools like funnel plots try to detect when the small, null studies are missing. The deeper habit is simpler: treat the absence of published failures as a question, not a reassurance.

    For those of us who build evidence reviews or cite best practices to clients, the discipline is to ask what is not on the page. A practice that looks well supported may simply be one whose failures were never written down. The most important studies in a review are sometimes the ones you cannot find.

    So here is my question: When you assess a body of evidence, how do you account for the studies that were run but never published, and has the missing research ever changed your read of what the evidence really says?