Stephen T’s Blog Spot

One of the biggest shifts in how evidence gets made is the rise of real-world data and real-world evidence. It began at the FDA and is spreading across health agencies and into evidence-based policy more broadly. The appeal is obvious: instead of slow, narrow, expensive clinical trials, use the mountain of data the system already generates, electronic health records, insurance claims, disease registries, wearables, to learn what actually works in the world. It is a powerful idea, and it is also where nearly every methods problem in this series shows up at once.

Start with a distinction the FDA is careful to make, and so should we. Real-world data is the data: information on health and care collected routinely, outside of any trial. Real-world evidence is the conclusion about whether something works, derived from analyzing that data. Those are not the same thing. Data is not evidence. The phrase real-world evidence is earned by what you do to the data, not granted by where the data came from. The entire question is the leap from one to the other.

The first reason that leap is hard is fitness for purpose. Real-world data was generated for care, billing, or operations, never to answer your research question. A diagnosis code exists to justify a payment, not to record clinical truth. So the first thing to ask, and the first thing the FDA asks, is whether the data is complete, accurate, and reliable enough, and whether it even captures the right population, exposure, and outcome for this question. A dataset can be perfectly fit for one question and useless for the next. A huge but ill-fitting dataset is not an asset. It is a confident-looking trap.

The second reason is deeper. Setting aside trials that build randomization into routine care, most real-world data is not randomized. Who received which treatment was decided by doctors and patients for reasons bound up with the outcome. So real-world evidence inherits every threat this series has worked through: selection bias, confounding, data that is missing in non-random ways, and gaps in who appears in the database at all. The FDA’s second question is precisely whether the study design can support a causal claim. And scale offers no rescue. A hundred million biased records give you an extremely precise wrong answer.

The field’s best response is to stop treating the data as a shortcut around a trial and start using it to emulate one. Target trial emulation, now central to thinking at the FDA and its counterparts abroad, forces you to specify the randomized trial you would have run, eligibility, treatment, outcome, follow-up, and then build that design within the data. The discipline surfaces the biases instead of burying them. Pragmatic trials go further, embedding real randomization inside ordinary care, buying a trial’s internal validity while keeping real-world reach. This is how raw data becomes credible evidence, and even then it rests, honestly, on assumptions about unmeasured confounding that cannot be fully tested.

None of this is a knock on real-world evidence. It answers questions trials cannot: long-term and rare outcomes, the full messy population that trials screen out, and situations where a trial would be impossible or unethical, all faster and at lower cost. Done well, it complements trials rather than replacing them. The point is narrower and sharper. Real-world and big describe the data. Only design and scrutiny turn it into evidence.

So when someone hands you real-world evidence, ask the two questions in order. Was the data fit to answer this question, and was the design strong enough to carry a causal claim? And when you produce it, earn the word evidence with the design, then go looking for the confounder that would overturn you.

So here is my question: As real-world evidence shows up in more decisions, how do you separate the genuinely well-designed studies from the ones trading on the size and the real-world label alone?