Stephen T’s Blog Spot

A blog aimed at issues only data scientists, data analysts, statisticians, evaluators, and researchers care about.

recent posts

about

I spend my time thinking about how the government can use research and evaluation methods to make evidence informed decisions and policy. It can be a struggle! That is why I need your thoughts. Join me, have a read, and make a comment. Its how we add to the evidence base,
  • Understanding the Risks of Early Detection in Health Screening

    Understanding the Risks of Early Detection in Health Screening

    Few ideas in health feel more obviously correct than this one: catch the disease early, and you will do better. It is intuitive, it is often true, and it has driven decades of screening campaigns. So let me say clearly at the start that screening saves lives for several conditions, and nothing here argues for abandoning it. The argument is narrower and more uncomfortable: screening is not always beneficial, it can do net harm, and the reflex to detect everything earlier deserves more scrutiny than it usually gets.

    Start with why early detection helps only sometimes. Finding a disease sooner changes the outcome only if acting sooner changes its course. For some diseases it does. For others, the timing of detection does not change where the story ends, and moving the diagnosis earlier only means the person lives longer as a patient, not longer overall.

    That hides a measurement trap that makes screening look better than it is. Three biases inflate its apparent benefit:

    1. Lead-time bias. Move the diagnosis date earlier and you lengthen survival measured from diagnosis, even if the date of death does not move at all. People appear to survive longer when all that changed was the starting line.

    2. Length-time bias. Screening preferentially catches slow-growing cases, because they sit in the detectable but silent stage far longer. The fast, aggressive cases tend to surface as symptoms between screening rounds. So screen-detected disease is a gentler, more survivable sample of the disease to begin with.

    3. Overdiagnosis. The extreme of length bias. Screening detects disease that never would have caused symptoms or death in the person’s lifetime. These patients cannot be helped, because there was nothing to prevent. They can only be harmed by the treatment that follows.

    This is why counting cancers caught early, or comparing survival rates, is the wrong yardstick. Both are inflated by the biases above. The honest measure is whether screening lowers mortality, ideally death from all causes, in a fair comparison. Optimizing the easy proxy, cases found, instead of the real goal, deaths prevented, is the same trap this series flagged with Goodhart’s law.

    The false-positive problem compounds this, and it is a base-rate problem in disguise. When a disease is rare, even an accurate test produces many more false alarms than true findings, so a positive result can mean far less than it seems. Across repeated rounds of screening, the chance of at least one false positive climbs steadily, and each one can bring anxiety, follow-up procedures, and biopsies that carry their own risks. False negatives, in turn, deliver false reassurance.

    The cautionary cases are real and well documented. When South Korea added thyroid ultrasound to routine screening, thyroid cancer diagnoses rose roughly fifteenfold over about two decades while deaths from thyroid cancer stayed flat, a textbook overdiagnosis epidemic that produced a wave of thyroid surgeries, each with real complications. Infant screening for neuroblastoma, tested in controlled studies in Germany and Canada, raised detection substantially but did not reduce advanced disease or mortality, and Japan ended its national program. The long debates over PSA testing and over mammography turn on these same trade-offs, which is why expert bodies have moved toward individualized decisions and still disagree about exactly when and how often to screen.

    None of this means early detection is a myth. It means screening is a medical intervention like any other, with benefits and harms to be weighed for each disease and test, not assumed. The classic checklist, from Wilson and Jungner at the World Health Organization in 1968, still holds: the condition should matter, there should be a recognizable early stage, the test should be acceptable, treatment should work better when started earlier, and the benefits should outweigh the harms and costs. Cervical and colorectal screening clear that bar convincingly. Others clear it only for certain ages or risk groups, and a few do not clear it at all.

    So yes, we should temper the demand. Early detection is a means, not an end. The end is less suffering and fewer deaths, and screening earns its keep only when it delivers them. The mature position is not screen everything or screen nothing. It is to ask, honestly and disease by disease, whether finding this earlier actually helps, and to tell people the whole story, including the harms, not only the reassuring half.

    So here is my question. When you see a screening program promoted with survival rates and cases caught early, do you ask the harder question of whether it actually lowers mortality, and at what cost?

  • Why Programs Succeed or Fail: The Mechanism Explained

    Why Programs Succeed or Fail: The Mechanism Explained

    “Does the program work?” sounds like the most basic question an evaluator can ask. It is also close to unanswerable as written, because the same program routinely succeeds in one place and fails in another, for reasons that a simple yes or no can never hold. The honest answer is almost always: it depends. Realist evaluation takes that “it depends” seriously and turns it into a method.

    The starting point, set out by Ray Pawson and Nick Tilley in 1997, is that programs do not work the in the same way a drug is imagined to work. A program does not cause an outcome by itself. It offers resources, or opportunities, and what produces the result is how people respond to that offer. That response is the mechanism, and whether people respond one way or another depends on their context. The same job-training program can lift employment for one group and do nothing for another, not because the program changed, but because it set off different reasoning in a different setting.

    This reframes the unit of analysis. Instead of asking for the average effect, realist evaluation asks for a configuration: in this context, what mechanism does the program trigger, and what outcome follows? The product of the evaluation is a set of these context-mechanism-outcome statements. In this context, for these people, this mechanism fired and produced this result; in that context, a different mechanism fired and produced something else. The aim is to explain the pattern, not just to detect it.

    Contrast that with the familiar experimental question. A clean trial treats the program as a black box and estimates its average effect on the people in the study. That is valuable, but it tells you whether the box worked on average, in this sample, not why, for whom, or whether it will work anywhere else. A program with a healthy average effect can still be useless or harmful for a sizable subgroup, and a program judged a failure may have worked well wherever the context was right. The average can hide the mechanism completely.

    The payoff for getting this right is transfer. When you know the mechanism and the conditions it needs, you have something you can carry to a new site and reason about in advance: this works when these conditions hold, so here is what to expect somewhere new. “The program works” does not travel. “This mechanism fires when these conditions are present” does. For a decision-maker weighing whether to scale or adopt, that is usually the more useful kind of knowledge.

    None of this is free. Realist evaluation is demanding. It requires a theory of the mechanisms before you start, data on context and not merely on outcomes, and the discipline to test and refine your configurations rather than narrate them. It is less tidy than a single headline number, and it resists the clean summary that sponsors often want. It also carries a real risk: without specifying what would count as disconfirming, a set of context-mechanism-outcome stories can slide into unfalsifiable explanation. The rigor lives in stating the conjectures clearly and putting them to the data.

    For those of us working in federal evaluation, this is not abstract. Programs run across wildly different sites, populations, and conditions, and a single effect averaged over all of them can be both technically correct and practically worthless. A program office deciding where to expand and where to pull back is rarely served by “it works.” It is served by knowing what works, for whom, under what conditions, and through what mechanism. That is the question worth the effort, even when it refuses to fit on one line.

    So here is my question. When you report that a program works, do you also specify for whom and under what conditions, or does the average effect quietly stand in for the whole story?

  • Avoiding Confirmation Bias: The Importance of Negative Cases

    Avoiding Confirmation Bias: The Importance of Negative Cases

    In qualitative analysis there is a quiet, dangerous pull. You form an interpretation early, often in the first few interviews, and from then on the data seem to cooperate. Each new account appears to confirm the pattern. The theme feels stronger with every quote you add to it. But a stack of confirming cases is weak evidence, because you were never really looking for anything else. The strength of a qualitative claim does not come from how many examples support it. It comes from how hard you looked for the ones that do not.

    That discipline has a name: negative case analysis. It means deliberately searching your own data for instances that do not fit the interpretation you are building, and then taking those instances seriously. It is one of the established techniques for qualitative credibility, named by Lincoln and Guba alongside prolonged engagement, triangulation, and peer debriefing. And it is the qualitative cousin of an idea I return to often: you learn far more from trying to break your account than from trying to confirm it.

    This is the direct antidote to confirmation bias in interpretive work. Left to its own devices, the mind notices what fits and quietly explains away what does not. Negative case analysis reverses the question. Instead of asking what supports my theme, you ask what would contradict it, and then you go looking to see whether that contradiction is sitting in your data already. The cases that do not fit are not noise to be smoothed over. They are often the most informative material you have.

    The mechanics are straightforward, even if the honesty they require is not. Once you have a candidate interpretation, you go back through the data hunting for the disconfirming instance: the participant who did the opposite, the account that cuts against the pattern, the exception that the tidy version of your finding would prefer to ignore. When you find one, you do not delete it. You change the theory. Either the interpretation widens to absorb the case, or it narrows and gains a boundary condition, this holds here, but not there. Each negative case either kills a claim or sharpens it. The grounded-theory tradition does this continuously through constant comparison, but any careful analysis can do it on purpose.

    Here is the part that surprises people. Hunting for disconfirmation makes your findings more credible, not less. A claim that has survived a real search for counterexamples is far stronger than one propped up by a dozen agreeable quotes. And the boundary conditions you uncover along the way, the places where the pattern holds and the places where it breaks, are frequently more useful than the pattern itself. Knowing where a finding stops applying is knowledge, not weakness.

    A note of honesty is required, though. This only works if you are genuinely willing to be wrong, which is harder than it sounds when you are facing a deadline and you like the story your data are telling. It is also not a license to throw out a well-supported interpretation the moment a single odd case appears. The discipline is to engage the exception, understand why it differs, and decide whether it revises the claim, bounds it, or truly overturns it. Writing down that reasoning, rather than burying the case, is itself part of the rigor.

    In evaluation, this has real teeth. Qualitative findings often drive recommendations, and recommendations move money and programs. So when a comfortable consensus forms, when the report is gliding toward everyone agrees the program is working, the right reflex is to ask who did not agree, and whether anyone went looking. The dissenting site, the participant who dropped out, the frontline staffer who pushed back: those are the negative cases, and they are usually where the real lessons are hiding. Actively seeking them is what protects you from a flattering conclusion that happens to be false.

    So here is my question. When you reach a confident qualitative finding, do you stop and search for the case that would break it, or does the search quietly end once the pattern feels strong enough?

  • Hindsight Is Not Insight

    Hindsight Is Not Insight

    After a program fails, the review almost writes itself. The warning signs were there all along. The decision looks negligent. Surely someone should have seen it coming. The trouble is that most of this clarity is manufactured. It appears only after you learn the outcome, and it quietly rewrites your memory of how uncertain things really were at the time.

    Psychologists call this hindsight bias, or creeping determinism. In a classic 1975 study, Baruch Fischhoff showed that once people are told how an event turned out, they raise their estimate of how predictable it had been, and they misremember their own earlier guesses as closer to the truth. Knowing the ending makes the whole story feel inevitable. People given the outcome inflated the probability they claimed they would have assigned beforehand, by a meaningful margin.

    There is a closely related trap. We judge the quality of a decision by how it turned out, rather than by what was known when it was made. Jonathan Baron and John Hershey demonstrated in 1988 that the same choice is rated as wise when it ends well and foolish when it ends badly, even when people are told the available information was identical and are instructed to ignore the result. A sound bet that loses still gets graded as a bad bet. The name for this one is outcome bias.

    This is an occupational hazard for anyone who evaluates programs and decisions after the fact, which is most of us. With the outcome in hand, a failed initiative looks obviously doomed and its leaders careless, while a success looks inevitable and its strategy brilliant. Both readings are distorted by the same mechanism. We end up praising luck and punishing sound judgment that happened to draw a bad card.

    The damage runs deeper than unfair reviews. We learn the wrong lessons. If every bad outcome is treated as a bad decision, we teach people to avoid reasonable risks and to manage appearances instead of making good calls. We also miss the dangerous cases, the good outcomes that came from poor reasoning and simple luck, which are exactly the ones likely to fail next time. Hindsight does not just misjudge the past. It corrupts what we carry forward.

    The fixes have to be deliberate, because the bias is automatic. Judge the decision by what was knowable at the time, and reconstruct the information and constraints the decision-makers actually faced. Hold apart two questions that hindsight fuses: was this a good decision, and did it turn out well? They are not the same. Where you can, write down predictions and reasoning before the outcome is known, so you are not rebuilding them later through the fog of the result. And use the debiasing move with the best evidence behind it: deliberately ask how things could have gone differently, and why a careful person might have chosen as they did.

    For those of us in evaluation, lessons-learned reviews and after-action reports are where this bias does its quietest work. The disciplined approach is to reconstruct the decision as it looked going in, to credit sound process even when results disappointed, and to resist the satisfying story in which everything that happened was always going to happen. The aim is to learn what was actually learnable, not to enjoy the false comfort of a tidy, inevitable past.

    So here is my question: When you review a project that went badly, do you separate the quality of the decision from the quality of the outcome, or does the result color the whole assessment?

  • Understanding Colliders in Statistical Models

    Understanding Colliders in Statistical Models

    There is a piece of advice that sounds responsible and is sometimes exactly wrong. When you worry that a comparison might be confounded, the reflex is to control for more variables, to put everything into the model just to be safe. But some variables do not remove bias when you control for them. They create it. They can manufacture a relationship that was never there.

    The troublemaker has a name: a collider. A collider is a variable that is a common effect of two others, two arrows pointing into it. When two things both influence a third, something strange happens if you hold that third thing fixed, by adjusting for it, stratifying on it, or restricting your sample to it. The two causes become artificially correlated, even if they had nothing to do with each other. Conditioning on the common effect opens a path that was supposed to stay closed.

    An example makes it concrete. Imagine a selective program that admits people who are strong on either a written test or an interview. Across all applicants, test and interview scores might be unrelated. But among those admitted, they will look negatively correlated: someone who got in with a mediocre test probably had a strong interview, and the reverse. The negative relationship is purely an artifact of selecting on admission, the collider. Nothing about the applicants changed. The sample did.

    This is not a classroom curiosity. The oldest version is Berkson’s paradox, described in 1946: study only hospitalized patients and two unrelated conditions can appear linked, because each one raises the chance of being hospitalized. A modern version is the obesity paradox. In the general population, obesity raises mortality, yet among patients who already have heart disease, it can appear protective. Heart disease is a collider, a common effect of obesity and other risk factors, and conditioning on it distorts the picture. Adjust for the wrong variable and a harm can look like a benefit.

    Here is what makes this genuinely hard. It is the mirror image of confounding. A confounder is a common cause of the treatment and the outcome, and you must adjust for it to get the right answer. A collider is a common effect, and you must not. The two can look identical in a dataset. The only way to tell them apart is to reason about the causal structure, which arrow points where, before you decide what goes into the model. The numbers alone will never tell you.

    This is why the comforting rule, control for everything you can measure, is wrong. Some of those variables are colliders. Others are affected by the treatment itself, which makes them colliders too. Adding them all does not buy safety; it can introduce bias that was not there before. And, as in a recent post, more data does not save you: a larger sample just pins down the distorted association more precisely.

    So the real move is conceptual, not statistical. Before you adjust for a variable, ask what causes it. If the treatment or the outcome, or the things that drive them, influence that variable, leave it out. Causal diagrams exist precisely to make these choices visible, marking which variables are confounders to control and which are colliders to avoid. The urge to control for more should give way to controlling for the right things.

    In evaluation, this hides in plain sight. We restrict samples to program completers, or to those who responded, or to sites that reported data, and we load models with covariates to look thorough. Every restriction and every control can remove bias or create it. Selecting on completion, response, or survival is conditioning on a collider, the same trap in a different coat.

    So here is my question: When you choose your control variables, do you reason about what causes what, or does the list come from whatever happens to be in the dataset?

  • Debunking Myths: Quantum Computing’s Role in Research

    Debunking Myths: Quantum Computing’s Role in Research

    A note before I begin. This post is a departure from my usual focus on research methods and evaluation. I am writing it for a simple reason: I find quantum computing genuinely fascinating, and the claims swirling around it and the social sciences have grown loud enough that I wanted to sort fact from fiction for myself. So treat this as a curious detour rather than a methods lesson, though the habit behind it is the same one I bring to everything here: be skeptical, check the evidence, and do not confuse a promise with a result.

    Start with the claim you have probably seen: that quantum computing is about to revolutionize how we analyze behavior, run surveys, or model society. That is fiction, at least for now. We are in what researchers call the NISQ era, noisy intermediate-scale machines, and a genuine quantum advantage over ordinary computers for practical problems has not been demonstrated. Current optimization experiments handle roughly twenty to thirty variables. Serious projections put useful applications somewhere around 2035 to 2040, and only if hardware keeps improving. This is real and exciting science, but it is not a tool you will run your next analysis on.

    There is a deeper obstacle that rarely makes the headlines. Even a mature quantum computer would have to get your data into it, and loading large, messy, high-dimensional social data is itself slow and costly, often costly enough to erase the very speedup that made the quantum approach attractive. Social data is the opposite of the clean, structured input these machines handle best. That mismatch, not just the qubit count, is why the analytics claims deserve caution.

    Here is a confusion worth clearing up, because it generates a lot of misplaced excitement. You will see references to ‘quantum models’ in psychology and decision research. These have almost nothing to do with quantum computers. Quantum cognition, developed by Jerome Busemeyer, Emmanuel Pothos, and others, borrows the mathematics of quantum probability, not the hardware, to model human judgment. The same math physicists use for noncommuting measurements neatly captures things like the conjunction fallacy and question order effects, the fact that asking A before B gives different answers than B before A. Readers of this series will recognize that from my post on question wording. It is a clever modeling framework that runs on an ordinary laptop. It is not evidence that your brain is a quantum computer, and it is not quantum computing.

    If there is one place quantum computing should already be on your radar, it is not analysis. It is security. A future quantum computer running Shor’s algorithm could break the encryption that protects almost all digital data today. That threat already has a name: ‘harvest now, decrypt later,’ where an adversary stores your encrypted data now to decrypt once the hardware exists. For anyone holding sensitive research or government data that must stay confidential for a decade or more, this is not hypothetical. In August 2024, after an eight-year process, the National Institute of Standards and Technology finalized the first post-quantum encryption standards, and federal systems are now on a migration timeline. That is the quantum story most relevant to our work, and it is about protecting data, not crunching it.

    None of this means quantum computing is irrelevant to social science forever. The plausible long-horizon uses are narrow: certain optimization problems, and simulating complex systems that classical machines struggle with. But narrow and long-horizon is the honest description, not revolution by next year. The pattern is the same one I warn about with any shiny method: the gap between what a technology can do in principle and what it can do for your problem today is usually large, and the burden is on the claim to close it.

    So that is my attempt to tease fact from fiction. The science is genuinely thrilling, the security implications are real and current, the analytics revolution is mostly not here, and the ‘quantum mind’ headlines are usually about borrowed math, not machines.

    So here is my question, especially for those who know this area better than I do. Where do you see quantum computing realistically touching our work first, and which claims have you learned to discount?

  • Why Big Data Can Mislead: The Importance of Sample Representativeness

    Why Big Data Can Mislead: The Importance of Sample Representativeness

    There is a comforting belief worth dismantling. When a dataset is huge, millions of records, we relax. Surely something that large is representative of the population. It is one of the most expensive intuitions in modern data work, because size and representativeness are not the same thing. A very large sample can be more confidently wrong than a small one.

    Start with the distinction that matters. Representativeness comes from the sampling mechanism, not from the size. A probability sample, in which every member of the population has a known, nonzero chance of being selected, earns the right to generalize and to quantify its own uncertainty. A nonprobability sample, people who opted in, an online panel, whatever an administrative system happened to record, has no such mechanism. Its representativeness is a hope, not a property.

    In 2018 the statistician Xiao-Li Meng made this precise, and unsettling. He showed that the error in a sample estimate depends on three things: how variable the outcome is, how much of the population you captured, and what he named the data defect correlation, the correlation between whether someone shows up in your data and what their answer would be. The sting is what this means as the sample grows: the bias does not shrink. With a very large sample, the random noise becomes tiny, so your confidence interval narrows to a sliver, but it narrows around the wrong number. You end up precisely, confidently mistaken. Meng put it memorably: the more the data, the surer we fool ourselves.

    The numbers are startling. Studying the 2016 United States election, Meng found that a correlation of about half of one percent between responding and which candidate a person favored was enough that a self-reported sample of more than two million carried the same accuracy as a genuine random sample of about four hundred. The rest of that sample bought almost nothing. The pattern returned during the pandemic: a survey run through Facebook, gathering roughly two hundred fifty thousand responses a week, produced worse estimates of vaccination rates than a small probability survey of about a thousand. Bigger lost to better.

    This should change how we hear the phrase big data. The flood of administrative records, app logs, opt-in panels, and scraped datasets is overwhelmingly nonprobability data. It is often enormous and almost always self-selected, which is precisely the recipe the paradox warns about. Size reassures us at the exact moment we should be most suspicious.

    So what do you do? When you can, use a probability design, because the known selection mechanism is what licenses the inference, and volume does not replace it. When you cannot, you must model your way out, with weighting and post-stratification, tools I have written about, which adjust the sample to resemble the population. But those corrections fix only the bias you can model with the variables you have, and they lean on assumptions you cannot fully test. They help. They do not turn a self-selected pile into a random sample.

    For those of us in evaluation, the lesson is practical. When someone hands you a massive dataset and offers its size as proof of its quality, ask the unglamorous question first. How did people end up in this data, and could that selection be related to what we are trying to measure? If it could, more rows will not rescue you. They may sink you faster, and with more confidence.

    So here is my question: When you are offered a very large dataset, do you interrogate how people selected into it, or does the sheer size tend to quiet the doubts it should raise?

  • Who Is This Evaluation Actually For? Utilization-Focused Evaluation

    We pour enormous effort into getting the methods right: the design, the analysis, the careful caveats. Then the report is delivered, politely thanked, and set on a shelf, where nothing happens to it. In evaluation, the most common failure is not a flawed method. It is irrelevance. A technically flawless study that changes no decision has failed at the one thing it was for. Michael Quinn Patton built an entire approach around that problem, and called it utilization-focused evaluation.

    Its premise fits in three words: intended use by intended users. That sounds too obvious to state, and it is routinely ignored. Most evaluations are aimed at an audience, which is an amorphous, anonymous entity, which is to say no one in particular. A report for everyone tends to be used by no one. Utilization-focused evaluation throws out the audience and replaces it with specific, named primary intended users and the specific decisions they actually have to make.

    This rests on what Patton called the personal factor, the part worth remembering. Across his research the pattern repeated: evaluations got used when a real, identifiable person or small group personally cared about the findings, and went unused when that person was missing. The lesson is concrete. The program is not a user. The program director who will brief the board in March on whether to renew the contract is a user. That gap is the gap between a report that moves something and a report that gathers dust.

    So in this approach you identify those people first, engage them early, and let their intended use drive every later choice: the questions you ask, the methods you pick, the timeline, even how you present results. Use, Patton argued, flows from clarity of purpose. His guiding question to a client is bracingly practical: what do you not know that, if you did know it, would change what you do? If a finding would not change any decision, it probably does not deserve the evaluation’s scarce attention.

    There are two bonuses. Involving users early builds ownership, and people act on what they feel ownership of. And there is what Patton named process use: simply taking part in a serious evaluation changes how people think about their program, sometimes before a single finding arrives. Methods stay flexible throughout, quantitative, qualitative, or mixed, chosen for what will be useful rather than for disciplinary habit.

    Now the honest tension. Getting close to the people who will use your work raises an obvious danger: that the evaluation drifts toward telling them what they want to hear, trading independence for relevance. Utilization-focused evaluation does not license that. Findings still have to be credible, or all you have achieved is the enthusiastic use of a flattering fiction. The skill is holding both at once, close enough to be used and independent enough to be trusted. Relevance without credibility is spin; credibility without relevance is the report on the shelf.

    So before you design the next study, ask who specifically will act on it, and what decision they face. Then design backward from that, while guarding the independence that makes the findings worth acting on in the first place.

    So here is my question: Before you start, do you name the actual person who will use the findings and the decision they need to make, and how do you stay useful to them without becoming captured by them?

  • Understanding Confirmation Bias in Research

    Understanding Confirmation Bias in Research

    Last time I argued that the heart of a credible causal claim is hunting for the explanations you would rather were not true. That is harder than it sounds, and the reason is the most human bias in all of research: we instinctively look for evidence that we are right, not evidence that we are wrong. It rarely feels like a bias. It feels like confidence.

    The cleanest demonstration is sixty years old. In 1960, the psychologist Peter Wason showed people the sequence 2, 4, 6, told them it followed a hidden rule, and asked them to discover the rule by proposing their own triples, learning only whether each one fit. Most quickly formed a hypothesis, often something like even numbers rising by two, and tested it with triples like 4, 6, 8 and 10, 12, 14. Each came back yes. Confidence climbed, they announced their rule, and they were wrong. The actual rule was simply any increasing numbers. They had never tried a triple that could fail, like 1, 2, 3 or 6, 5, 4, so they never learned their rule was too narrow. The confirming examples felt like progress. They were a trap.

    That is the whole problem in miniature. Piling up evidence consistent with your idea tells you very little, because the evidence that would actually test the idea is the evidence that could contradict it. Raymond Nickerson, in a landmark 1998 review, showed how far this reaches: confirmation bias shapes what evidence we go looking for, how we read ambiguous evidence, even what we later remember. In research it decides which studies we cite, which analyses we run and quietly keep, which interview quotes jump out as themes, and which inconvenient cases we explain away. None of it feels like cheating. It feels like the data agreeing with us.

    The antidote is old, and it connects to the falsifiable theory of change I wrote about earlier. Karl Popper’s point was that you learn the most from trying to refute an idea, not to support it. So the question to put to your own work is not what supports my conclusion, but what would show it is wrong, and have I genuinely gone looking for that? A finding you only tried to confirm has barely been tested. A finding that survived a serious attempt to break it has earned some trust.

    Here is the uncomfortable twist. Willpower is a weak defense. In follow-up studies, simply telling people to consider disconfirmation did not reliably make them better at it; we cannot easily out-stubborn our own minds in the moment. That is why the durable fixes are structural, not attitudinal. Pre-register the hypothesis and analysis before you see the data, so you cannot later redefine success to fit the result, the same guard against p-hacking I have written about. Assign someone whose job is to argue the opposite, a red team or adversarial reviewer. Analyze blinded where you can, so you do not know which number is the one you hoped for. The aim is to build disconfirmation into the process, so being right does not depend on you being heroically objective on a Tuesday afternoon.

    This only gets sharper as we lean on AI assistants, which will cheerfully elaborate whatever hypothesis we hand them. A tool that agrees with you is comfortable and dangerous. Ask it for the strongest case against your finding, not just for support.

    So here is my question: Before you analyze, do you name what would change your mind, and who in your process is actually responsible for trying to prove the conclusion wrong?

  • Making the Case Without a Control Group: Contribution Analysis

    You are asked the question every program eventually faces: what difference did it make? But there is no control group. You could not randomize. A dozen other things were happening at the same time, the economy, other programs, the simple passage of time. The counterfactual methods I have written about, matching, regression discontinuity, difference-in-differences, all need a comparison you simply do not have. And yet giving up on the causal question is not an option, because the people who funded the work still need to know whether it worked. This is the situation contribution analysis was built for.

    Start with a shift in the question. Most rigorous causal methods chase attribution: how much of the outcome did the program cause, measured against a world where it never happened? Contribution analysis, developed by the evaluator John Mayne, asks something humbler and often more honest: did the program make a meaningful difference, as one cause among several? You are not claiming sole credit, and you are not producing a precise percentage. You are building a credible case that the program mattered, alongside everything else that was going on.

    The backbone of that case is the theory of change from the last post. Mayne’s logic runs roughly like this. You can reasonably conclude a program contributed when four things hold. Its theory of change is plausible. The activities were actually carried out, not just planned. The chain of expected results in between is confirmed by evidence, not assumed. And the other plausible influences on the outcome have been examined and either ruled out or given their share of the credit. Meet those conditions and you have something defensible. The product is not an effect size. It is a reasoned, evidence-backed argument, what Mayne calls a contribution story.

    Now the part that separates the method from a press release. The whole exercise lives or dies on that fourth condition, the alternative explanations. Anyone can assemble the evidence that flatters their program and arrange it into a tidy narrative. That is not analysis; it is marketing with footnotes. What makes contribution analysis rigorous is the obligation to go hunting for the rival explanations and to confront them head on. Maybe the outcome improved because the broader economy did. Maybe a second initiative was running in parallel and deserves the credit. Maybe the people served would have improved on their own. A contribution story that never names a single rival it had to argue against is not a finding. It is a testimonial.

    Be honest about what you get and do not get. Contribution analysis yields a plausibility judgment and a stated level of confidence, not the clean number a randomized experiment delivers. Because it leans on reasoning rather than a counterfactual, it is only as strong as the evidence you bring and the seriousness of your search for alternatives. Many evaluators now pair it with process tracing, which sets explicit tests for how strong a piece of evidence needs to be before it counts. None of that turns it into an experiment. It is what you do responsibly when an experiment was never on the table.

    And in real program work, it usually is not. So the discipline is this. When you cannot randomize, do not retreat to ‘we cannot say,’ and do not leap to ‘it obviously worked.’ Build the theory of change, gather evidence on every link in it, and then spend most of your energy on the explanations you would prefer were not true. The credibility of the whole claim rests there.

    So here is my question: When you cannot build a counterfactual, how do you decide a program genuinely contributed, and how hard do you actually look for the alternative explanations that would undercut your story?

  • Bad Idea, or Badly Delivered? The Theory of Change and Logic Models

    Almost every federal program and grant proposal asks for a logic model or theory of change. And most of the time, one gets drawn, made to look tidy, approved, and never opened again. It becomes paperwork, a diagram that satisfied a reviewer. That is a waste, because used as intended, a theory of change is the most useful analytic tool in evaluation. The trick is to stop treating it as a picture and start treating it as a hypothesis.

    First, what it is. A theory of change maps the causal chain of a program: the inputs and activities, what you put in and do; the outputs, what those activities produce; and the outcomes you expect to follow, in the short, intermediate, and long term. A logic model is the schematic version, a sequence of if-then steps; a theory of change goes further, spelling out the mechanism and why each link should hold. In both, the meaning lives in the arrows, not the boxes: the arrows are claims that doing this will cause that.

    And that is the reframe. A theory of change is a falsifiable hypothesis about how change happens. Every arrow is an assertion that could turn out to be false. Its worth is not that it looks complete or comprehensive. Its worth is that it is explicit enough to be wrong, and therefore explicit enough to test. Evaluation researchers have even argued for building a deliberately falsifiable logic model, with measurable benchmarks at each intermediate step. A theory of change you could never disprove is not a theory. It is decoration.

    Here is where it pays off, and it is the heart of the matter. Suppose the final outcome does not move. With only a before-and-after number, that is a dead end: it did not work, and you cannot say why. But with an explicit theory of change, a disappointing result splits into two very different diagnoses. The first is implementation failure: you never actually delivered the activities, or they did not produce the intended outputs, so the theory was never really tested. The second is theory failure: you delivered everything as planned, the outputs appeared, and the expected change still did not follow, which means a causal assumption was simply wrong.

    This distinction, which goes back to Edward Suchman and Carol Weiss in the field’s foundations, matters because the two failures demand opposite responses. Implementation failure says the idea may be fine, fix the delivery. Theory failure says the delivery was fine, rethink the idea. Confuse them and you will scrap a sound program for being poorly run, or pour resources into faithfully delivering something that was never going to work. Without a theory of change you cannot tell them apart; with one, a null result becomes a map of where the program broke.

    The most valuable parts of the model are the assumptions hiding between the boxes. If we train the staff, they will use the training. If clients show up, their behavior will change. If the report is delivered, someone will act on it. Those unstated links are where programs quietly die, and the discipline of a theory of change is dragging them into the open where they can be examined and measured.

    A caution: a clean diagram can mislead as easily as it clarifies. It can imply a tidy, linear world when reality has feedback loops, context, and surprises, and it can lend false confidence just by looking organized. The model is a hypothesis to hold loosely and revise as evidence arrives, not a promise to defend.

    So build it as a working tool, not a grant ornament. Let it tell you what to measure, the intermediate links and not only the final outcome, and let it commit you in advance to what success and failure would look like.

    So here is my question: When a program underdelivers, how do you tell implementation failure from theory failure, and does your logic model actually let you measure the steps in between?

  • Groups Are Not Big People: The Ecological Fallacy

    We are swimming in aggregate data. Averages by county, by ZIP code, by store, by team, by region. It is everywhere, it is cheap, and it is tempting to read it as if it described the people inside each group. That temptation has a name, the ecological fallacy, and it is one of the oldest and most stubborn errors in quantitative work.

    The fallacy is drawing a conclusion about individuals from a correlation measured across groups. The two can disagree completely. The classic demonstration came from the sociologist William Robinson in 1950. Using 1930 census data, he examined the relationship between being foreign-born and being illiterate, computed two ways. Across the 48 states, the correlation was negative: states with more immigrants tended to have lower illiteracy. But across individuals it was positive: a foreign-born person was, on average, somewhat more likely to be illiterate. The two numbers did not just differ in size. They pointed in opposite directions.

    How can that happen? Because the groups differ from each other in ways that have nothing to do with the individuals inside them. Immigrants in 1930 tended to settle in states that already had well-educated native-born populations. So states with many immigrants looked highly literate, not because the immigrants were literate, but because of everyone else in those states. The state average was a fact about states. It was never a fact about immigrants, and treating it as one inverted the truth.

    This is a cousin of Simpson’s paradox, which I wrote about earlier, but it is its own distinct trap. Simpson’s paradox is about a trend reversing when you pool or split the same units. The ecological fallacy is about changing the unit of analysis entirely, from the group to the person, and assuming the relationship survives the trip. It usually does not, at least not at the same strength, and sometimes not even in the same direction.

    Why does this matter now? Because aggregated data is often all we can get, and it is seductive precisely because it is so available. A dashboard says regions with more of some service have better outcomes. A report shows schools with a certain feature score higher. A vendor claims customers in zip codes with trait X buy more of product Y. Each is a statement about groups. The instant you restate it as a claim about a person, a region, a school, a customer, you may have crossed into fiction, and a confident, numerical one at that.

    None of this means aggregate data is worthless. Often it is the only data we have, and it can answer group-level questions perfectly well. The error is not in using it. The error is in quietly swapping the question, asking about people and answering about places, or asking about places and answering about people. The two questions have two different answers, and only sometimes do they agree.

    So the discipline is to be ruthless about the unit of analysis. Before you accept any correlation, ask what was actually counted: individuals, or groups of them? If the data are aggregated and the claim is about individuals, the honest response is not to believe it and not to dismiss it, but to treat it as a hypothesis that can only be tested with individual-level data. The map is informative, but it is not the people, and the average of a place tells you surprisingly little about anyone who lives there.

    So here is my question: Where have you seen a group-level pattern confidently sold as a fact about individuals, and how do you keep the unit of analysis straight when the only data on hand is aggregated?

  • Just Above the Line, Just Below It: The Regression Discontinuity Design

    We have been working through selection bias, and last time I covered propensity-score matching, which rebuilds comparable groups but only on the characteristics you managed to measure. There is a design that, where it applies, does something matching cannot: it balances even the things you did not measure. It is beloved in the methods literature, and it is also, as you may suspect, harder to use in practice than its fans let on. It is called regression discontinuity.

    The setup is specific. A program is assigned by a strict cutoff on some continuous score. Students above a test score get the scholarship; households below an income line get the benefit; patients above a risk threshold get the drug. Regression discontinuity compares the units just above the line to those just below. The student one point over the scholarship cutoff and the student one point under it are, for practical purposes, the same student. One got the award and one did not, for no reason that matters. That arbitrary line behaves like a coin flip in a narrow band, and the jump in the outcome right at the cutoff is the estimated effect.

    This is why methodologists prize it. Near the threshold, the two groups are alike on everything, including the traits you never recorded, because nobody can perfectly control landing just above versus just below an arbitrary line. That is the as-if-randomness matching cannot give you. The idea goes back to a 1960 study of merit awards by Thistlethwaite and Campbell, and it is now regarded as one of the most credible ways to estimate an effect without an actual experiment.

    So why the skepticism? Because the conditions that make it work are demanding, and three of them bite hard.

    First, you need a real cutoff. The design only applies where treatment is genuinely assigned by a sharp threshold on a measured number. Most programs are not assigned that way, so you cannot simply decide to use regression discontinuity. You can only use it when a law or rule happens to have created a clean line for you. It is an opportunity you find, not a tool you deploy.

    Second, the answer is stubbornly local. Even when it works perfectly, it tells you the effect only for the units right at the threshold, the marginal scholarship winner, the barely-eligible applicant. It says little about people well above or below the line. You buy excellent internal validity at the cost of a narrow, sometimes awkwardly specific, conclusion.

    Third, and most damaging in practice, people can manipulate the line, and they do so precisely where the stakes are high enough to bother. If anyone can nudge their score across the threshold, the units on either side are no longer comparable, and the magic evaporates. This is not hypothetical: researchers have documented teachers bumping student test scores over performance thresholds, and students arranging their credits to clear a scholarship cutoff. There is a standard check, the density test, which looks for a suspicious pile-up of cases on the lucky side of the line, but two-directional manipulation can slip past it.

    Add the fact that the credible comparison lives in a thin slice of data around the cutoff, so you need a large sample to have enough cases near the line, and you can see the shape of the problem. Regression discontinuity is close to a randomized experiment when the cutoff is real, ungamed, well populated with data, and you are content with a local answer. That is a lot of conditions. Its beauty is genuine, and so is its narrowness.

    So here is my question for the group. Have you found a setting where a real cutoff handed you a clean natural experiment, or have the thresholds in your work turned out too soft, too gamed, or too sparse to lean on?

    Regression discontinuity design

    We have been working through selection bias, and last time I covered propensity-score matching, which rebuilds comparable groups but only on the characteristics you managed to measure. There is a design that, where it applies, does something matching cannot: it balances even the things you did not measure. It is beloved in the methods literature, and it is also, as you may suspect, harder to use in practice than its fans let on. It is called regression discontinuity.

    The setup is specific. A program is assigned by a strict cutoff on some continuous score. Students above a test score get the scholarship; households below an income line get the benefit; patients above a risk threshold get the drug. Regression discontinuity compares the units just above the line to those just below. The student one point over the scholarship cutoff and the student one point under it are, for practical purposes, the same student. One got the award and one did not, for no reason that matters. That arbitrary line behaves like a coin flip in a narrow band, and the jump in the outcome right at the cutoff is the estimated effect.

    This is why methodologists prize it. Near the threshold, the two groups are alike on everything, including the traits you never recorded, because nobody can perfectly control landing just above versus just below an arbitrary line. That is the as-if-randomness matching cannot give you. The idea goes back to a 1960 study of merit awards by Thistlethwaite and Campbell, and it is now regarded as one of the most credible ways to estimate an effect without an actual experiment.

    So why the skepticism? Because the conditions that make it work are demanding, and three of them bite hard.

    First, you need a real cutoff. The design only applies where treatment is genuinely assigned by a sharp threshold on a measured number. Most programs are not assigned that way, so you cannot simply decide to use regression discontinuity. You can only use it when a law or rule happens to have created a clean line for you. It is an opportunity you find, not a tool you deploy.

    Second, the answer is stubbornly local. Even when it works perfectly, it tells you the effect only for the units right at the threshold, the marginal scholarship winner, the barely-eligible applicant. It says little about people well above or below the line. You buy excellent internal validity at the cost of a narrow, sometimes awkwardly specific, conclusion.

    Third, and most damaging in practice, people can manipulate the line, and they do so precisely where the stakes are high enough to bother. If anyone can nudge their score across the threshold, the units on either side are no longer comparable, and the magic evaporates. This is not hypothetical: researchers have documented teachers bumping student test scores over performance thresholds, and students arranging their credits to clear a scholarship cutoff. There is a standard check, the density test, which looks for a suspicious pile-up of cases on the lucky side of the line, but two-directional manipulation can slip past it.

    Add the fact that the credible comparison lives in a thin slice of data around the cutoff, so you need a large sample to have enough cases near the line, and you can see the shape of the problem. Regression discontinuity is close to a randomized experiment when the cutoff is real, ungamed, well populated with data, and you are content with a local answer. That is a lot of conditions. Its beauty is genuine, and so is its narrowness.

    So here is my question: Have you found a setting where a real cutoff handed you a clean natural experiment, or have the thresholds in your work turned out too soft, too gamed, or too sparse to lean on?

  • Matching on a Single Number: Propensity-Score Matching

    Last time I described selection bias: when people choose into a program, the participants and non-participants differ before anything happens, so a naive comparison measures the people, not the program. Randomization would solve it, but often we cannot randomize. So how do you rebuild a fair comparison from data where the groups were never balanced? The most popular answer is propensity-score matching.

    The obvious idea would be to match each participant with a non-participant who looks just like them, same age, same prior earnings, same education, same everything, so that the only difference left is the program. With one or two characteristics that works. With twenty, it collapses: you will almost never find someone who matches on all of them at once. There are too many dimensions and not enough people.

    In 1983, Paul Rosenbaum and Donald Rubin showed a way around the wall. You do not have to match on all the characteristics directly. You can collapse them into a single number, the propensity score, the estimated probability that a given person would have ended up in the program, based on their observed characteristics. You calculate it, usually with a model that predicts treatment from the covariates, then match a participant to a non-participant who had the same probability of participating. Their visible traits may differ in detail, but on the whole they were equally likely to be in the program.

    The reason this works is quietly elegant. Among people who were equally likely to enroll, who actually did enroll looks, on the measured characteristics, close to random. So comparing a treated person to an untreated person with the same propensity score compares like with like, and the comparison starts to behave like the randomized experiment you could not run. You can check it, by producing the table of characteristics for both groups after matching and confirming they finally look alike. The same logic supports related moves: stratifying by the score, or weighting by it.

    Now the limitation, which is the whole reason last week’s post came first. A propensity score can only be built from the characteristics you measured. It balances age, earnings, and education because you have them; it can do nothing about what you never recorded. And in a voluntary program, the biggest driver of both enrolling and succeeding is often something soft and unmeasured, like motivation. Randomization would have balanced that too, silently, because the coin flip ignores everything. Propensity-score matching cannot, because it only knows what is in your data. Two groups can be perfectly balanced on every measured variable and still differ on the one that mattered most.

    This is why the method rests on an assumption you cannot test: that you measured everything driving both selection and outcome. It is the same shape of problem as the missing-data assumption from earlier this month, untestable from the data alone, and it calls for the same honesty. The strongest analyses pair the matching with a sensitivity analysis, asking how strong an unmeasured confounder would have to be to overturn the result. If it would take an implausibly powerful hidden factor, the finding is robust. If a modest one would do it, be nervous.

    So propensity-score matching is not a substitute for randomization, and anyone who sells it as one is overreaching. What it is, is a disciplined, transparent way to approximate a fair comparison when randomizing was never an option, with its assumptions laid out where reviewers can see them. It converts the vague worry “these groups are different” into the sharper, more answerable question “different on what, and did we measure it?”

    So here is my question: When you use matching or weighting to compare non-randomized groups, how do you argue that you captured the variables that drive selection, and do you stress-test for the ones you might have missed?

  • Did the Program Work, or Did the Right People Join?

    Picture a voluntary job-training program. A year later, the people who enrolled are earning more than the people who did not. Success, right? Maybe. But ask a harder question first: who signs up for a training program? Often the more motivated, the more employable, the people already on their way up. If the participants were going to do better anyway, then comparing them to non-participants does not measure the program. It measures the kind of people the program attracted. This is selection bias, and it is the single biggest threat to any claim that something worked.

    The core problem is comparability. To say a program caused an outcome, you need two groups that were alike in every relevant way except for the program, so that any later difference can be pinned on it. A randomized experiment delivers that by assigning people to treatment or control by chance. Randomization is powerful precisely because the coin flip does not care who you are; over enough people, the groups end up balanced on everything, the things you measured and the things you never thought to. The two groups are interchangeable at the start, so a difference at the end means something.

    Selection bias is what you get when that coin flip is missing. In the real world, people are rarely assigned to programs at random. They volunteer, they are referred, they qualify, they opt in. Each of those routes is a filter, and the filter is usually related to the very outcome you care about. The motivated enroll in the training. The health-conscious join the gym. The schools with resources adopt the new curriculum. So the treated group and the untreated group differ before the program ever starts, and that head start contaminates every comparison you make afterward.

    This is a close cousin to confounding, but it has its own character. Confounding is usually described as some background variable influencing both treatment and outcome. Selection bias is the version that operates through how people come to be in your groups at all, whether through their own choices or someone else’s sorting. And it does not only happen at enrollment. It can sneak in later, through who stays in the study, who responds, who remains in the data. Anywhere a non-random filter stands between the full population and the people you end up analyzing, selection bias can enter.

    What makes it so dangerous is that the data look complete and the analysis looks clean. You have a treated group, an untreated group, an outcome, and a tidy difference between them. Nothing in the spreadsheet warns you that the two groups were never alike. The bias is not a flaw in your calculation. It is baked into who is in the rows, and a more careful regression on the same biased groups will simply give you a more precise wrong answer.

    The instinct to compare participants to non-participants is so natural that it is everywhere, in program reports, marketing claims, and much observational research now sailing under the banner of real-world evidence. Sometimes randomization is impossible or unethical, and an observational comparison is all we have. That is legitimate. But it obligates us to take selection seriously rather than wish it away, and to ask, every time, whether the people we are comparing were ever truly comparable.

    Next time I will get to the most popular tool for fighting back, propensity-score matching, which tries to rebuild comparable groups after the fact. For now, the discipline is the question itself.

    So here is my question for the group. When you see a before-and-after or a participant-versus-nonparticipant comparison, what is the first thing you check to decide whether the two groups were genuinely alike to begin with?

    Selection bias

    Picture a voluntary job-training program. A year later, the people who enrolled are earning more than the people who did not. Success, right? Maybe. But ask a harder question first: who signs up for a training program? Often the more motivated, the more employable, the people already on their way up. If the participants were going to do better anyway, then comparing them to non-participants does not measure the program. It measures the kind of people the program attracted. This is selection bias, and it is the single biggest threat to any claim that something worked.

    The core problem is comparability. To say a program caused an outcome, you need two groups that were alike in every relevant way except for the program, so that any later difference can be pinned on it. A randomized experiment delivers that by assigning people to treatment or control by chance. Randomization is powerful precisely because the coin flip does not care who you are; over enough people, the groups end up balanced on everything, the things you measured and the things you never thought to. The two groups are interchangeable at the start, so a difference at the end means something.

    Selection bias is what you get when that coin flip is missing. In the real world, people are rarely assigned to programs at random. They volunteer, they are referred, they qualify, they opt in. Each of those routes is a filter, and the filter is usually related to the very outcome you care about. The motivated enroll in the training. The health-conscious join the gym. The schools with resources adopt the new curriculum. So the treated group and the untreated group differ before the program ever starts, and that head start contaminates every comparison you make afterward.

    This is a close cousin to confounding, but it has its own character. Confounding is usually described as some background variable influencing both treatment and outcome. Selection bias is the version that operates through how people come to be in your groups at all, whether through their own choices or someone else’s sorting. And it does not only happen at enrollment. It can sneak in later, through who stays in the study, who responds, who remains in the data. Anywhere a non-random filter stands between the full population and the people you end up analyzing, selection bias can enter.

    What makes it so dangerous is that the data look complete and the analysis looks clean. You have a treated group, an untreated group, an outcome, and a tidy difference between them. Nothing in the spreadsheet warns you that the two groups were never alike. The bias is not a flaw in your calculation. It is baked into who is in the rows, and a more careful regression on the same biased groups will simply give you a more precise wrong answer.

    The instinct to compare participants to non-participants is so natural that it is everywhere, in program reports, marketing claims, and much observational research now sailing under the banner of real-world evidence. Sometimes randomization is impossible or unethical, and an observational comparison is all we have. That is legitimate. But it obligates us to take selection seriously rather than wish it away, and to ask, every time, whether the people we are comparing were ever truly comparable.

    Next time I will get to the most popular tool for fighting back, propensity-score matching, which tries to rebuild comparable groups after the fact. For now, the discipline is the question itself.

    So here is my question: When you see a before-and-after or a participant-versus-nonparticipant comparison, what is the first thing you check to decide whether the two groups were genuinely alike to begin with?

  • Fill the Holes, Then Stress-Test Them: Multiple Imputation and Sensitivity Analysis

    In a previous post I argued that missing data is rarely random, that deleting or averaging it away quietly biases your results, and that you usually cannot prove whether the gaps are the benign kind or the dangerous kind. That is a bleak place to stop, so here is the repair. It comes in two moves: fill the holes honestly, then test how much your conclusion depends on the guesses.

    The first move is multiple imputation, an idea from the statistician Donald Rubin. The instinct most people have is to fill each blank with a single best guess, a predicted value or an average. The trouble is that a single guess pretends you know something you do not, and your software then treats that invented number as if it were real data, reporting a precision you have not earned. Multiple imputation refuses the pretense. Instead of filling each gap once, it fills it many times, drawing a range of plausible values from a model built on everything else you observed. You end up with several complete datasets that agree on what you measured and differ exactly where you were guessing.

    You then analyze each completed dataset normally and pool the results using what are called Rubin’s rules. The pooled estimate is just the average across the datasets. The clever part is the uncertainty. The pooled variance combines two things: the ordinary uncertainty within each dataset, and the variation between the datasets, the degree to which your answer wobbled as the guesses changed. That second piece is the honesty. It is the cost of not having observed the data, written directly into your confidence interval, where a single imputation would have hidden it.

    Multiple imputation is valid under the safe and the moderate cases, MCAR and MAR, and you can strengthen it by including auxiliary variables, things that help predict either the missing value or the fact that it is missing. Adding them makes the at-random assumption more plausible and shrinks the bias if it is not quite true.

    But recall the catch from last time: you cannot prove the data are missing at random. Multiple imputation assumes it. So a careful analysis does not stop there. It asks the second question: what if the assumption is wrong, and the missing values are systematically different from what the model expects?

    That is the job of sensitivity analysis, and the most intuitive version is the tipping point. You deliberately push the imputed values for the missing cases away from the at-random guess, making the dropouts worse off, say, by a little, then more. At each step you rerun the analysis. If your result survives even large, pessimistic departures, it is robust. If a tiny nudge flips it from significant to null, it was fragile all along, resting on an assumption you could never test. Reporting that tipping point, how far reality would have to depart from at-random to overturn your finding, is among the most honest things an analyst can do. It is now routine in well-run clinical trials, precisely because lives ride on the answer.

    Put the two moves together and you have a workflow with integrity. Impute many times to use all the information you have and to carry the uncertainty you cannot escape. Then stress-test the conclusion against the possibility that what you could not see was different from what you could. The goal is never to pretend the holes were not there. It is to show, out loud, exactly how much they could matter.

    So here is my question: When you handle missing data, do you go past imputation to test how fragile your conclusion is, and have you ever found that the answer hinged entirely on the cases you could not observe?

  • The Holes in Your Data Are Not Random: Missing Data Mechanisms (MCAR, MAR, MNAR)

    Every real dataset has holes. People skip the sensitive question, drop out of the study, or are never measured at all. A sensor fails, a record is incomplete, a field is blank. And most of the time, without quite deciding to, we handle those holes in one of two ways: we delete the rows that have them, or we fill the gaps with an average. Both feel harmless. Both can quietly wreck your conclusions. This post is about why; the next will be about what to do instead.

    The key insight, formalized by the statistician Donald Rubin in 1976, is that what matters is not how much data is missing but why it is missing. He described three mechanisms, and they could not be more different in their consequences.

    The first is missing completely at random, or MCAR. Here the missingness has nothing to do with anything, observed or unobserved. A tube of blood is dropped, a page is lost in the mail. The missing values are just a random subset of all the values. This is the safe case, and it is also the rarest. If your data really are MCAR, deleting the incomplete rows costs you sample size but does not bias the answer.

    The second is missing at random, or MAR, one of the most confusing names in statistics, because it is not random at all. MAR means the missingness depends on things you did observe, but not on the missing value itself. Suppose younger people are less likely to report income, but within any age group, whether they answer has nothing to do with how much they earn. The gap is patterned, but the pattern is explained by data you have, age, so you have a fighting chance to correct for it.

    The third is the dangerous one: missing not at random, or MNAR. Here the missingness depends on the very value that is missing. The highest earners decline to state their income precisely because it is high. The sickest patients drop out of the trial precisely because they are doing badly. The information you need to fix the problem walked out the door with the people who left. Nothing in your remaining data can tell you what you lost.

    Now the trap. Both default moves assume the safe case and betray you in the others. Deleting incomplete rows, which software often does automatically, is unbiased only if the data are MCAR. Under MAR or MNAR, the people who remain are a skewed subset, and your clean complete-case analysis is confidently wrong. Mean imputation is worse: filling every blank with the average shrinks the natural variation, weakens real correlations, and fabricates false precision. As one summary put it, if you do not consciously choose a method, you have chosen deletion by default.

    Here is the part that should keep you honest. You usually cannot prove which mechanism you are facing. You can rule out MCAR by checking whether the people with missing data differ on the things you did measure. But the line between MAR and MNAR depends on the values you never saw, so it cannot be settled by the data alone. It is, in the end, an assumption you must argue for, not a fact you can test.

    So the amount missing is the wrong thing to fixate on. A trivial fraction of MNAR can bias a result badly, while a large amount of MCAR mostly just costs precision. The first question about any hole in the data is not how big it is, but why it is there.

    So here is my question: When you hit missing data, do you stop to ask why it is missing before you decide how to handle it, and what tells you whether the gap is benign or dangerous?

  • Can Statistics Rescue a Biased Sample? Survey Weighting and MRP

    In another post I argued that a giant sample drawn from a skewed frame is still wrong, and that size cannot save it. That raises the obvious question. If you are stuck with an imperfect sample, and most of us usually are, can statistics fix it after the fact? The honest answer is: often, and sometimes impressively, but only up to a hard limit worth understanding.

    Start with the workhorse, weighting. Suppose your sample has twice as many young people as the population and half as many older ones. You assign each respondent a weight that pulls the sample back into line with known population totals, counting underrepresented groups for more and overrepresented groups for less. Anchored to solid benchmarks, usually census figures, this post-stratification corrects a sample that is lopsided on characteristics you can measure. When you know the population only one variable at a time, a technique called raking nudges the weights until they match every margin at once.

    The more powerful modern approach is a mouthful, multilevel regression and poststratification, mercifully shortened to MRP. Instead of only reweighting, you build a model that predicts the outcome from a person’s characteristics, then use it to estimate the answer for every demographic cell, and finally reassemble those cells in the exact proportions of the real population. The multilevel part quietly stabilizes the thin cells, where you have few respondents, by borrowing strength from the rest of the data.

    How well can this work? Consider the field’s most striking example. In 2012, researchers ran an opt-in election poll on the Xbox gaming platform. The sample was about as unrepresentative as you can imagine: roughly nine in ten respondents were men, and most were under thirty. The raw numbers were useless. Yet after MRP, the adjusted estimates tracked the professional poll aggregators and the actual result. A sample of gamers, properly modeled, measured the national electorate. MRP is now a standard tool, used for district-level forecasts and built into major media models. Even the doomed 1936 magazine poll could likely have been rescued, because it had quietly collected how people voted in the prior election, the kind of variable that makes adjustment work.

    So statistics can carry a biased sample a long way. Now the catch, which is the whole point. These methods can only correct for things you can measure and match to the population, and they all rest on one assumption: that within each group you adjust for, the people you have are a fair stand-in for the people you are missing. Weight by age, sex, and region, and you are betting that a missing person resembles, on the outcome, the respondents of the same age, sex, and region. If the people your frame never reached differ in some way you did not measure, no weighting scheme can recover them. The information simply is not there.

    There is a second, quieter cost. Heavy weighting throws away precision. When a handful of rare respondents must each stand for many, your effective sample shrinks and your uncertainty grows, even if the headline number looks fine. Extreme weights are a red flag, not a triumph.

    For those working with administrative data, opt-in panels, and convenience samples, the practical bottom line is this. Weighting and MRP can turn an imperfect sample into a defensible estimate, but only with good population benchmarks and a credible case that what is missing is captured by what you measured. The method is part mathematics, part honesty about that assumption.

    So here is my question: When you weight or model your way from a skewed sample to a population estimate, how do you decide whether the variables you adjusted for are enough, and how do you disclose who your numbers still cannot speak for?

  • A Big Sample of the Wrong People: Coverage Error and the Sampling Frame

    In 1936, a magazine ran the largest election poll the world had ever seen. The Literary Digest mailed out around ten million ballots and tallied roughly two and a half million that came back. On that mountain of data, it confidently predicted that Alf Landon would defeat Franklin Roosevelt. Roosevelt then won in one of the largest landslides in American history. The magazine’s reputation never recovered, and within two years it was gone.

    What went wrong is one of the most important lessons in all of survey research, and it has nothing to do with sample size. The Digest drew its names from telephone directories, automobile registrations, club rosters, and its own subscriber list. In 1936, in the depths of the Depression, telephones and cars were luxuries. So the list systematically overrepresented the well off, who leaned toward Landon, and missed the poorer voters who carried Roosevelt. The poll did not have too few people. It had the wrong people, and adding more of them would not have helped.

    This is coverage error, and it is distinct from the more familiar worry about who responds. Coverage error happens earlier, at the moment you choose the list you will sample from, what researchers call the sampling frame. If that frame does not match the population you actually care about, some people have no chance of being selected at all. Every later step can be flawless, the sampling random, the response rate high, the analysis clean, and the result will still be biased, because the bias was baked in before the first ballot went out.

    Here is the part that should unsettle anyone who works with data today. Size does not fix it. A biased frame sampled lightly and a biased frame sampled massively give you the same skewed answer, just with more decimal places of false confidence. The Literary Digest had two and a half million responses. That same year, George Gallup predicted the right winner from a sample a fraction of the size, because he chose it to mirror the country rather than to be large. Representativeness beat raw volume, and it was not close.

    That lesson lands harder now than it did then, because we are surrounded by enormous, effortless data that comes from skewed frames. An online survey reaches the people who are online and willing to click. An app’s usage data describes the people who already use the app. Social media sentiment captures the views of the people who post. Each of these can deliver millions of records and still quietly exclude whole segments of the population you mean to understand. Big data does not announce who is missing. You have to ask.

    For those of us who collect and analyze data, the discipline is to start with two questions before admiring any number. Who is in the frame, and who can never appear in it no matter how much data we gather? A customer list leaves out the customers you lost. A portal survey leaves out people who cannot reach the portal. A convenience sample leaves out everyone inconvenient. None of that is cured by a bigger pull from the same source. It is cured only by fixing the frame, or by being honest about exactly whom your findings do and do not describe.

    The seductive thing about a huge dataset is that it feels authoritative simply by being huge. The Literary Digest felt that way too. The number that matters is not how many people you reached, but whether the people you could reach look like the people you are trying to learn about.

    So here is my question: When you assess a data source, how do you probe who is missing from the frame entirely, and have you seen a large, impressive dataset quietly misrepresent the population it claimed to cover?

  • Three Bearings to a Single Point: Triangulation

    The word comes from navigation. A sailor cannot fix position from a single landmark; one bearing tells you the direction to the lighthouse, not where you are. Take a bearing on a second landmark, and a third, and the lines cross at one point. That crossing is your location. No single sighting could have given it.

    Social scientists borrowed the idea and the name. In 1978, Norman Denzin brought triangulation into research as a deliberate strategy: study the same thing from more than one vantage point, because any single method carries its own blind spots. He laid out four kinds. Data triangulation uses multiple sources, different people, times, and places. Investigator triangulation uses more than one researcher. Theory triangulation interprets findings through more than one framework. Methodological triangulation uses more than one method, interviews alongside records alongside a survey. The logic: if several approaches that fail in different ways all point to the same conclusion, that conclusion is hard to dismiss.

    That last phrase, fail in different ways, is the whole point, and it is where triangulation is often misunderstood. The strength does not come from piling on more data. It comes from combining sources whose weaknesses are independent. A survey can be distorted by what people are willing to admit. Administrative records can be distorted by how they were entered. Interviews can be distorted by who agreed to talk. But those distortions push in different directions, so when the three still converge, the agreement means something. Stacking three sources that share the same bias just gives you the same error three times, and a false sense of confidence.

    Now the part even seasoned researchers skip. We treat triangulation as a search for agreement, and feel the method has failed when the sources disagree. The methodologist Sandra Mathison argued the opposite in 1988. Convergence is only one possible outcome; sometimes the sources are inconsistent, and sometimes they flatly contradict each other, and those are not failures. They are findings. When your survey says the program is working and your interviews say people are quietly working around it, the disagreement is the most interesting thing in the room. It tells you the simple story is wrong, and points at where to look next.

    For those of us in evaluation, this is daily bread. We rarely get one clean measure of whether a program worked. We get outcome data, case files, site visits, and the accounts of the people involved, each partial. Triangulating them does two jobs. When they line up, we can make a contribution claim with real confidence, because the evidence would have broken in different places if we were wrong. When they diverge, we have found the seam in the story, where the official numbers and the lived experience part ways, and that seam is usually where the real learning is.

    A caution, though. Triangulation is a plan, not a label you paste on at the end. You have to choose sources that are genuinely independent, decide in advance what convergence and divergence would each mean, and resist the urge to quietly discard the source that spoils the agreement. The discarded source is often the honest one.

    So the rule I keep is this. Never trust a single bearing. Convergence earns confidence; divergence earns insight; and the finding that survives three different ways of being wrong is the one worth building on.

    So here is my question: When your data sources have disagreed, did you treat it as a problem to reconcile or a finding to chase, and what did the disagreement end up teaching you?