Top 10 Tips for Crafting Effective Survey Questions

By Mary Agnes Welch

Let’s be honest – we’ve all answered our share of confusing, boring, vaguely irritating survey questions in our lives. Heck, we’ve even written some of those. So, we’ve come up with a few rules we live by at Probe, rules that might help you next time you’re jotting down ideas for our quarterly omnibus survey:

1. No one understands or cares about your thing to the degree you do. You have expert knowledge, a deep interest and likely a unique vocabulary for your thing. Average people don’t live in that bubble with you. In fact, prior to taking your survey, they probably haven’t spent one minute thinking about your issue or product. They are grappling with a topic in ten minutes that you’ve been thinking about for ten years. Keep that in mind and make sure you’re respecting where people are at.

2. Ask questions Sesame Street-style. Clear, simple, conversational questions are always more effective, more engaging and more useful. Start with the basics and work your way up. When in doubt, think about how you’d ask your granny the question.

3. Keep it short. No, shorter than that. Nope, it’s still too long. Ten minutes max. Figure out what you really need to know and what’s just a nice-to-know. For every question, think – how will I use the responses to this? Every single question should propel you to action or strategy. If it doesn’t, drop it.

4. Focus on getting information, not giving it. A survey isn’t your chance to educate the world about your thing N=600 people at a time. That’s what ads are for. Long, technical descriptions and detailed preambles to questions turn off respondents, risk biasing responses and defeat the purpose of a survey: to listen and genuinely understand people and their opinions in good faith.

5. Ask people questions they can actually answer. The last thing you want is people abandoning a survey half-way through because they get frustrated and confused by the questions. So don’t ask people to pretend to be PhDs in public policy or experts in urban planning, or to think profoundly about every micro-decision they make at the grocery store. Instead, ask questions they can legitimately answer – what they like, what they value, how they act, what their priorities are.

6. Don’t ask why. Surveys are best at answering the “what” – what people value, what they do, what their satisfaction levels are, what they agree with. Surveys are bad at asking people “why”, in part because people are bad at explaining why they hold a certain view or behave a certain way. Save the “why” for a face-to-face discussion in a focus group.

7. Ask questions the way your opponent or competitor would. It’s really hard to check your own biases and not introduce them, subtly or overtly, into the structure of a question, its wording or the answer options. A biased or leading question does no one any good – it gets respondents’ backs up and it nets you bad data. So try to craft a question that even your most strident opponent or competitor would think was fair.

8. Be realistic about sub-pops. Yes, it might be valuable to know how low-income BIPOC women who live in the suburbs think or feel about a certain issue, but that is a tiny proportion of the population. Only a very big and costly survey would be able to poll enough people to find a critical mass of low-income suburban BIPOC moms, so be realistic about how much you can legitimately slice and dice the data with any statistical reliability. One rule of thumb is you need a sample size of 100 in any given group to be reasonably confident in the results.

9. Nix open-ended questions. Sometimes an open-ended, in-their-own-words style question is perfect. Mostly, though, they are really disappointing and a pain in the butt. People rarely offer deep and useful thoughts while they’re taking a survey on their phone in the grocery store line-up. They often don’t answer the question that’s asked. And open-ended questions can kill the momentum and flow of a survey, and often prompt people to abandon it altogether.

10. Consider the Unsure option. A good survey gives people a way forward with every question, and typically that includes an “unsure” option, especially if it’s a legit option. For example, people probably aren’t unsure whether they had breakfast that morning, but they may very well be unsure of whether they agree with a carbon tax. Sometimes, we want to force respondents into a choice, but more often than not an Unsure option makes sense.

Mary Agnes Welch is a principal with Probe Research. She joined the firm in 2016 following a career as an award-winning politics and public policy journalist at the Winnipeg Free Press. A graduate of Columbia University’s journalism program, Mary Agnes builds on her rich experience covering public policy to capture and communicate the underlying research story, focusing on the insights that matter most. Clients benefit tremendously from Mary Agnes’ engaging approach and her significant skills as a focus group moderator and facilitator, as she uncovers what citizens really think about complex issues.