This site focuses on all things related to Research and Evaluation
My Latest Posts
• • •
- The Variable You Should Not Control ForThere is a piece of advice that sounds responsible and is sometimes exactly wrong. When you worry that a comparison might be confounded, the reflex is to control for more… Read more: The Variable You Should Not Control For
- Quantum Computing and Social Science: Fact and FictionA note before I begin. This post is a departure from my usual focus on research methods and evaluation. I am writing it for a simple reason: I find quantum… Read more: Quantum Computing and Social Science: Fact and Fiction
- A Bigger Sample Is Not a Better OneThere is a comforting belief worth dismantling. When a dataset is huge, millions of records, we relax. Surely something that large is representative of the population. It is one of… Read more: A Bigger Sample Is Not a Better One
- Who Is This Evaluation Actually For? Utilization-Focused EvaluationWe pour enormous effort into getting the methods right: the design, the analysis, the careful caveats. Then the report is delivered, politely thanked, and set on a shelf, where nothing… Read more: Who Is This Evaluation Actually For? Utilization-Focused Evaluation
- The Bias That Feels Like Being Right: Confirmation BiasLast time I argued that the heart of a credible causal claim is hunting for the explanations you would rather were not true. That is harder than it sounds, and… Read more: The Bias That Feels Like Being Right: Confirmation Bias
- Making the Case Without a Control Group: Contribution AnalysisYou are asked the question every program eventually faces: what difference did it make? But there is no control group. You could not randomize. A dozen other things were happening… Read more: Making the Case Without a Control Group: Contribution Analysis
- Bad Idea, or Badly Delivered? The Theory of Change and Logic ModelsAlmost every federal program and grant proposal asks for a logic model or theory of change. And most of the time, one gets drawn, made to look tidy, approved, and… Read more: Bad Idea, or Badly Delivered? The Theory of Change and Logic Models
- Groups Are Not Big People: The Ecological FallacyWe are swimming in aggregate data. Averages by county, by ZIP code, by store, by team, by region. It is everywhere, it is cheap, and it is tempting to read… Read more: Groups Are Not Big People: The Ecological Fallacy
- Just Above the Line, Just Below It: The Regression Discontinuity DesignWe have been working through selection bias, and last time I covered propensity-score matching, which rebuilds comparable groups but only on the characteristics you managed to measure. There is a… Read more: Just Above the Line, Just Below It: The Regression Discontinuity Design
- Matching on a Single Number: Propensity-Score MatchingLast time I described selection bias: when people choose into a program, the participants and non-participants differ before anything happens, so a naive comparison measures the people, not the program.… Read more: Matching on a Single Number: Propensity-Score Matching
- Did the Program Work, or Did the Right People Join?Picture a voluntary job-training program. A year later, the people who enrolled are earning more than the people who did not. Success, right? Maybe. But ask a harder question first:… Read more: Did the Program Work, or Did the Right People Join?
- Fill the Holes, Then Stress-Test Them: Multiple Imputation and Sensitivity AnalysisIn a previous post I argued that missing data is rarely random, that deleting or averaging it away quietly biases your results, and that you usually cannot prove whether the… Read more: Fill the Holes, Then Stress-Test Them: Multiple Imputation and Sensitivity Analysis
- The Holes in Your Data Are Not Random: Missing Data Mechanisms (MCAR, MAR, MNAR)Every real dataset has holes. People skip the sensitive question, drop out of the study, or are never measured at all. A sensor fails, a record is incomplete, a field… Read more: The Holes in Your Data Are Not Random: Missing Data Mechanisms (MCAR, MAR, MNAR)
- Can Statistics Rescue a Biased Sample? Survey Weighting and MRPIn another post I argued that a giant sample drawn from a skewed frame is still wrong, and that size cannot save it. That raises the obvious question. If you… Read more: Can Statistics Rescue a Biased Sample? Survey Weighting and MRP
- A Big Sample of the Wrong People: Coverage Error and the Sampling FrameIn 1936, a magazine ran the largest election poll the world had ever seen. The Literary Digest mailed out around ten million ballots and tallied roughly two and a half… Read more: A Big Sample of the Wrong People: Coverage Error and the Sampling Frame
- Three Bearings to a Single Point: TriangulationThe word comes from navigation. A sailor cannot fix position from a single landmark; one bearing tells you the direction to the lighthouse, not where you are. Take a bearing… Read more: Three Bearings to a Single Point: Triangulation
- Looking Where the Light Is Good: The Streetlight effect and the McNamara FallacyThere is an old joke about a man searching for his lost keys under a streetlight. A passerby asks where he dropped them. “Over in the park,” he says, “but… Read more: Looking Where the Light Is Good: The Streetlight effect and the McNamara Fallacy
- From Two Points to Two Trends: Controlled Interrupted Time SeriesIn another post I described difference-in-differences, the design that estimates a policy’s effect by comparing the change in a treated group to the change in an untreated one. Today I… Read more: From Two Points to Two Trends: Controlled Interrupted Time Series
- When One Place Changes and Another Does not: Difference-in-DifferencesIn a separate blog, I wrote about synthetic controls, a clever way to build a comparison group when you do not have one. Today, its simpler and far more common… Read more: When One Place Changes and Another Does not: Difference-in-Differences
- Triangulation Is Not a Validity StampLast time I argued that mixed methods is really about integration. There is one word that often stands in for that integration and quietly does a great deal of unearned… Read more: Triangulation Is Not a Validity Stamp
- Can This Program Be Evaluated?Here is a scene familiar to anyone who has done this work. A program has run for two years. The funder wants an evaluation, a contract is awarded, and a… Read more: Can This Program Be Evaluated?
- Real-World Data Is Not Real-World EvidenceOne of the biggest shifts in how evidence gets made is the rise of real-world data and real-world evidence. It began at the FDA and is spreading across health agencies… Read more: Real-World Data Is Not Real-World Evidence
- Write the Trial You Wish You Could RunThe most famous cautionary tale in observational research goes like this. For years, study after study found that women on hormone replacement therapy had healthier hearts, and it was widely… Read more: Write the Trial You Wish You Could Run
- A Good Instrument Is Hard to FindAcross several posts on causal inference, propensity-score matching, regression discontinuity, target trial emulation, I kept hitting the same wall. Each method could adjust for the confounders you had measured, but… Read more: A Good Instrument Is Hard to Find
- Who Is This Evaluation Actually For?We pour enormous effort into getting the methods right: the design, the analysis, the careful caveats. Then the report is delivered, politely thanked, and set on a shelf, where nothing… Read more: Who Is This Evaluation Actually For?
- A Good Instrument Is Hard to FindAcross several previous posts on causal inference, propensity-score matching, regression discontinuity, target trial emulation, I kept hitting the same wall. Each method could adjust for the confounders you had measured,… Read more: A Good Instrument Is Hard to Find
- Understanding Publication Bias and the File-Drawer ProblemWe live in the age of evidence-based everything: evidence-based policy, evidence-based practice, evidence-based design. When someone says “the research shows,” it is meant to end the argument. But a quiet… Read more: Understanding Publication Bias and the File-Drawer Problem
• • •
Stephen T’s Blog Spot 