Stephen T’s Blog Spot

A dataset lands on your desk, large and clean, and someone asks the natural question: is it good? That question has no answer as posed. Data is not good or bad in the abstract. It is fit, or unfit, for a particular use. The same dataset can be excellent for one question and worthless, even misleading, for another. Quality is not a property of the data. It is a relationship between the data and the purpose you bring to it.

This is not a personal opinion; it is the settled definition in the field. Data quality is standardly defined as fitness for use, quality judged by the data consumer and the task at hand. And it has two very different faces. There are the intrinsic virtues everyone thinks to check, accuracy, completeness, consistency, currency. And there are the contextual ones that actually decide whether the data can answer your question: relevance to what you are asking, coverage of the right population, capture of your construct, timeliness for your decision. A dataset can pass every intrinsic test and fail the contextual ones completely. Accurate is not the same as useful.

The clearest way to see this is with the kind of data we are increasingly handed. Robert Groves, a former director of the Census Bureau, drew a useful line between designed data, gathered deliberately to answer a question, and organic data, the exhaust of transactions and operations: case files, billing records, eligibility systems, service logs. Organic data accumulates whether or not anyone intends to analyze it. Most of what now arrives on our desks is organic: a found object, not a designed instrument, built to run a program, not to evaluate one.

That origin is exactly where the trouble hides, because the categories in operational data encode operational needs, not research constructs. A label like served might mean a case was opened, not that a person received help. A field gets filled in when it triggers a payment and left blank when it does not, so the missing values follow the workflow rather than chance. The population is everyone the system happened to touch, which quietly excludes everyone who never entered it, often the very group you need to see. Definitions drift as policies and software change, so a sharp trend can be the fingerprint of a form redesign rather than a change in the world. None of these show up in an accuracy check. The data can be flawless about the wrong thing.

Underneath all of it is a familiar idea in new clothing. Whether a dataset fits your use is really a question about the gap between what the data actually records and what you are trying to learn, which is a matter of construct validity, a theme this series has visited before. Having the data is not the same as being able to answer the question, and treating the two as interchangeable is one of the quietest ways an analysis goes wrong, because it goes wrong before a single number is computed.

For those of us in federal research and evaluation, the pull toward found data is strong and often sensible. It is cheaper, faster, and already sitting there, and agencies increasingly expect us to use it rather than field something new. But the convenience conceals the risk. When a program office says to just use the administrative data, the honest first move is not to run the analysis. It is to interrogate the data: what was this built to do, and does that match what we need it to tell us? Reaching for the data that happens to be available, rather than the data the question requires, is the streetlight problem in a modern form.

Deciding whether a dataset is actually fit for your purpose is a skill, not a hunch, and it can be done systematically. That is where the next post will go. For now, the shift in mindset is the whole point. Stop asking whether the data is good, and start asking what it is good for.

So here is my question. When a convenient dataset arrives, do you first ask what it was built to do and whether that matches your question, or does its size and cleanliness stand in for fitness?