Stephen T’s Blog Spot

A survey question asks a lot more than it appears to. To answer well, a respondent has to interpret what you meant, search memory for the relevant information, weigh it into a judgment, and then map that judgment onto the response options you offered. That is real cognitive work, and we assume that respondents are willing to do all of it, on every question, to the end of the questionnaire. Often they are not. And when they are not, they will look to take a shortcut.

The name for that shortcut is satisficing, a term the survey methodologist Jon Krosnick brought into this field in 1991. Instead of doing the full work to produce the best answer, the optimal answer, the respondent settles for one that is merely good enough to seem reasonable. They are not lying, and usually not careless in any dramatic way. They are conserving effort, exactly as people do on any demanding task, and the survey quietly absorbs the cost in the form of worse data.

It shows up in familiar patterns, once you know to look. A satisficing respondent tends to agree with whatever a question asserts, a habit called acquiescence. They pick the first response option that sounds acceptable rather than reading to the end of the list. On a long grid of rating items, they choose one column and run straight down it, a pattern called straightlining or nondifferentiation. They reach for the midpoint, or they say they do not know when they actually hold a view. Each of these is a plausible-looking answer produced with very little of the thinking the question was designed to capture.

Krosnick’s real insight was in explaining when this happens. Satisficing is not a character flaw in your sample. It is the predictable product of three things: how hard the task is, how able the respondent is, and how motivated they are. Make the question harder, or the respondent more tired, distracted, or indifferent, and the shortcut becomes more tempting. That means satisficing is partly under your control, because you built the task. A confusing item, a fatiguing grid, a questionnaire that runs too long, each one raises the price of a good answer.

There is a trap here that deserves its own warning. Satisficing can make your data look better while making it worse. When respondents straightline a battery of items, their answers become highly consistent, and internal-consistency reliability, the statistic many take as a sign of a good scale, goes up. But that consistency is an artifact of the shortcut, not evidence that the items measured anything well. It is a reminder of an earlier post in this series: reliability is not validity, and a number that rises for the wrong reason is worse than no number at all.

The encouraging part is that the same three levers can reduce it. Lower the task difficulty with simpler, clearer items, shorter grids, and an instrument no longer than the purpose requires. Support motivation by explaining why the survey matters and respecting the respondent’s time. And design out the easy escapes: think twice before offering a blanket no-opinion option that invites people to skip the work, and vary the order of response options so primacy does not do the answering. You cannot force optimizing, but you can lower the cost of it.

For those of us who collect survey data for federal clients, this is not a fine point. The estimates we deliver, and the decisions built on them, assume the answers reflect what respondents actually think and do. Satisficing quietly breaks that assumption, and it does so most where burden is highest and interest is lowest, which is often exactly the population a program most needs to hear from. Treating response quality as something we design for, rather than assume, is part of the craft.

So here is my question. When you build an instrument, do you design it to make a careful answer easy to give, or do you unintentionally reward the respondent for taking the shortcut?