Stephen T’s Blog Spot

In my previous post from earlier today I argued that data is never good in the abstract, only fit or unfit for a particular use. That leaves the practical question. Given a dataset and a purpose, how do you actually decide whether the first can serve the second? The answer is not a feeling about data quality. It is an interrogation, and these are the six questions I would ask in order.

1. What was this built to do? Establish provenance before anything else: who created the data, for what operational purpose, under what incentives, and what has happened to it since. That history, the lineage, tells you where the definitions came from and what pressures shaped them. A field that triggers a payment is recorded with great care. A field that no one uses is filled in casually. You cannot judge fitness without the biography.

2. Does the variable mean what my construct means? Take each variable you plan to lean on and write down the definition your question requires, then find the definition the system actually uses. Compare them explicitly. Served, active, completed, and eligible all have operational meanings that rarely match research constructs. The gap between the recorded field and the intended concept is where analyses quietly go wrong.

3. Who is in this data, and who could never be? Coverage decides what population your findings can describe. An administrative system contains the people it touched, which excludes those who never applied, were screened out, or dropped away before the record was created. Ask what the denominator really is. If the people missing from the frame differ systematically from the people in it, no amount of analysis on the records you have will tell you about the ones you do not.

4. Why are the values missing? Missingness in operational data is rarely random. Fields go blank because a workflow branched, a requirement did not apply, or a caseworker was busy. That mechanism matters more than the missingness rate, because it determines whether the gaps are ignorable or a source of bias, a point this series has made before. A dataset that is 5 percent missing for a reason related to your outcome is more dangerous than one that is 30 percent missing at random.

5. Is it timely enough for the decision it must inform? Data has a shelf life set by the question. Check when the data was collected, how long the lag runs, and whether the definitions or systems changed midstream. A break in a trend is often a form redesign or a policy change rather than a change in the world. And an answer that arrives after the decision has been made is not fit for that use, however accurate it is.

6. Can I write this down so someone else can check it? Document what you learned: the source, the operational purpose, the definitions, the coverage, the missingness mechanism, the known limits, and the uses the data can and cannot support. The field has converged on this idea in the form of the dataset datasheet, which records a dataset’s motivation, composition, collection process, and recommended uses. If you cannot produce that record, you do not yet know your data well enough to defend a finding drawn from it.

Two things stand out about this list. First, fitness for use is a verdict about a pairing, this data for that question, so it must be reassessed whenever either changes. Data blessed as fit for a performance report is not thereby fit for an impact evaluation. Second, none of these questions requires advanced statistics. They require curiosity and a willingness to ask uncomfortable things about a convenient dataset before it becomes the foundation of a finding. That is what data governance means in practice for a researcher: not a compliance exercise, but knowing your data well enough to say what it can honestly support.

So here is my question. Could you write a page documenting the provenance, definitions, coverage, and limits of the dataset your current analysis depends on, and if not, what would it change to find out?