Stephen T’s Blog Spot

You build a model and it fits beautifully. The line threads through nearly every point, the R-squared is high, and each bump in the data is captured. It feels like success. Very often it is the opposite. A model that fits the data in front of you more closely can predict new data worse, and the tighter that in-sample fit, the more suspicious you should be.

The culprit is overfitting. Any real dataset is a mix of signal, the true underlying pattern you care about, and noise, the random quirks specific to this particular sample. A flexible enough model will happily fit both. But the noise will not repeat in the next sample, so fitting it does not just fail to help, it actively hurts. Add enough parameters and you can drive the error on your data to zero, fitting every wrinkle. At that point the model has not learned the pattern. It has memorized the sample, and it will fall apart on data it has not seen.

Behind this sits a tradeoff worth naming. A model that is too simple misses real structure and is wrong in a consistent way; statisticians call that bias. A model that is too complex chases the noise and swings wildly from sample to sample; that is variance. Push complexity down and bias grows; push it up and variance grows. Good modeling lives in the balance, flexible enough to catch the signal, disciplined enough to ignore the noise. Complexity is never free, and more of it is not better.

The practical consequence is the important part. Because in-sample fit is inflated by exactly this problem, you cannot judge a predictive model on the data used to build it. The fit statistic there is close to a vanity metric. The only honest test is how the model performs on data it has never seen. So you hold out a portion of the data, build the model on the rest, and check it against the untouched part. Or you use cross-validation, training repeatedly on most of the data and testing on the piece left out, then averaging for a steadier estimate. Out-of-sample performance is the number that speaks to the future.

This is really the prediction cousin of two ideas this series has already visited. A model tuned to fit one sample perfectly is close kin to a result that only appears because someone tried enough specifications until something clicked. Both mistake the quirks of a particular sample for a general truth. And a very large dataset is no protection if you let the model grow more complex as the data grows, because a big enough model can memorize a big enough sample just as easily.

There is a subtle way the discipline fails even when people mean well. The held-out test is only honest if it stays untouched. If you check it, adjust the model, check it again, and repeat, the test set quietly becomes part of the training, and your out-of-sample estimate is no longer out of sample. The safeguard is to keep a genuinely sealed holdout for one final look, and to resist the urge to peek.

For those of us in research and evaluation, this matters more every year, because predictive models are spreading through the work: risk scores, targeting and needs models, and machine-learning tools sold as decision aids. When a vendor or a colleague reports how accurate a model is, the first question is simple. Accurate on what data? If the number comes from the same data the model was trained on, it tells you almost nothing about how the tool will behave in the field. Ask for out-of-sample performance, ideally on a different time period or site, which is the external-validity question wearing new clothes.

So here is my question. When you judge a model, do you insist on seeing how it performs on data it was never allowed to learn from, or does a beautiful fit on the training data quietly do the convincing?