Stephen T’s Blog Spot

You compare this year’s survey to last year’s, and the numbers have moved. Or you compare a phone sample to an online one, and they disagree. The natural reading is that something changed in the world, or that one of the samples is off. But there is a quieter explanation that is easy to overlook: you changed how you asked. The mode of a survey, whether it is administered by phone, on the web, in person, or on paper, is not a neutral pipe that delivers the question and returns the answer untouched. It shapes the answer.

The reason is that each mode puts the respondent in a different situation. The most consequential difference is whether another person is present. When an interviewer is listening, on the phone or across a table, people edit their answers toward what looks acceptable, so questions about sensitive matters draw more flattering responses than the same questions answered alone on a screen. This is the social desirability effect, and the mode turns it up or down. Self-administration, with no one watching, tends to produce more candid answers to sensitive questions, and it also changes other behavior: without an interviewer to keep the pace, respondents are freer to rush, skip, or take mental shortcuts.

Presentation matters too. On a screen or a page you see all the options at once and can reread them; on the phone you hear them in sequence and have to hold them in memory. That difference nudges which options get chosen, with a tendency toward the last options heard in a spoken list and the first ones seen in a visual one, though the evidence on exactly when this happens is mixed. Long lists of choices are simply harder to process by ear, which invites the kind of satisficing shortcuts this series has discussed. Even the way people use a rating scale shifts, with interviewer modes drawing more agreement and telephone respondents reaching more often for the extremes.

Put all this together and you arrive at the uncomfortable part: mode differences can look exactly like real differences. There is a striking illustration in the research literature. A study that established the instrument measured the same construct across modes, and controlled for who selected into each, still found that people interviewed face to face reported systematically better psychological functioning than people answering on the web. The gap was not a measurement artifact in the usual sense, and not a difference in who responded. It was the mode itself, most likely social desirability from the interviewer’s presence. The way the question was delivered moved the answer.

This is why switching modes over time is so dangerous. Many surveys have moved from phone to online in recent years, to cut costs and to fight the long decline in response rates. That is often the right call. But if last year was phone and this year is web, a shift in the results can be the mode change rather than a change in the population, and you can report a trend that is really a measurement discontinuity. It is the comparability problem from the measurement-invariance post in a new guise: before you compare, you have to be sure you measured the same way.

Mixed-mode designs, now common for the same cost and response-rate reasons, help coverage but add a twist. The people who answer by one mode may differ from those who answer by another, and the mode also nudges their answers, so mode and selection effects get tangled and are hard to separate. You cannot make any of this vanish, but you can manage it. Hold the mode constant when your aim is comparison over time, or change it deliberately and run an overlap study that measures the mode effect so you can adjust for it. Design questions to work equivalently across modes rather than optimizing each in isolation. And when you must mix modes, treat the mode as a variable in the analysis instead of pretending it is invisible. The one thing not to do is switch quietly and read the resulting change as news.

So here is my question for the group. When your numbers move between two surveys, do you rule out the possibility that only the mode changed before you conclude that the world did?