Stephen T’s Blog Spot

You calculate a 95 percent confidence interval and describe it the natural way: there is a 95 percent chance the true value lies inside it. Almost everyone reads it like this, and it is wrong. Not because the arithmetic failed, but because you asked one framework of statistics for an answer only the other framework can give. Behind a surprising share of misread numbers is a quiet collision between two different ways of thinking about uncertainty.

Start with the older and more dominant one. Frequentist statistics treats the unknown quantity you care about, the true effect, the real population value, as a fixed number. What is random is your data, drawn as one sample out of many you could have collected. So probability, in this world, describes how your procedure behaves over the long run. A 95 percent confidence interval means that if you repeated the whole study over and over, about 95 percent of the intervals you built that way would contain the true value. It says nothing about this one interval. The truth is either inside it or not; the confidence lives in the method, not in the particular answer sitting in front of you.

The other framework turns the picture around. Bayesian statistics treats the unknown quantity as itself uncertain, something you hold a degree of belief about, and treats the data as the fixed thing you actually observed. You begin with a prior, an explicit statement of what you believed before seeing the data, you update it with the evidence through Bayes’ rule, and you end with a posterior, a full probability distribution over the unknown. From that you can build a credible interval, and it means exactly what people want a confidence interval to mean: given your data and your model, there is a 95 percent probability the value lies inside. It is the direct answer to the direct question.

This is why the confusion is so universal. People are natural Bayesians. When we see a result, what we want to know is how probable the truth is, given what we saw, and that is a Bayesian question. But the standard tools of the last century, p-values and confidence intervals, are frequentist, and they answer a different question about the long-run behavior of a procedure. So we take a frequentist number and read a Bayesian meaning into it. The p-value becomes the probability the null is true; the confidence interval becomes the probability the truth is inside. These misreadings are not sloppy so much as wishful: the answer we actually wanted, projected onto a tool that cannot supply it.

Neither framework escapes without a cost, and it is worth being fair about both. The Bayesian approach gives you the direct probability, but only by requiring a prior, and where that prior comes from is a real question. State it well and it encodes genuine knowledge; state it carelessly and a strong prior can swamp weak data and smuggle your assumptions into the answer. The frequentist approach refuses the prior and the subjectivity that comes with it, and pays by being unable to make any probability statement about the parameter itself, only about the procedure. With a neutral prior the two often produce nearly identical intervals: the numbers can coincide while the meaning does not.

The practical point is not to pick a side. It is to know which question your number answers, and to resist reading one framework’s result as though it were the other’s. If a decision truly needs a probability about the thing itself, how likely it is that this program helps, and by how much, that is a Bayesian question and deserves a Bayesian method with a stated, defensible prior. If you want guaranteed long-run error properties and no reliance on a prior, frequentist tools are right, as long as you read them as statements about the procedure. The error is never choosing one; it is using one and interpreting it as the other.

So here is my question. When you report a confidence interval or a p-value, are you clear, to yourself and your audience, about which question it actually answers, and which one everyone secretly wants it to?