Magpi Tip: Open-Ended Questions in Form Design
Mobile data collection is getting more and more popular, as smartphones and tablets fall in price and increase in capabilities. Magpi has been supporting mobile data collection for 10+ years, and Magpi staff have been doing it for decades – not surprisingly we’ve heard a lot of questions about form design. Some are related to technology, while others are just related to good practice in data collection, even on paper. For this Magpi Tip, we’ll look at the topic of open-ended versus closed-ended (e.g. multiple-choice) questions.
When to use open-ended questions versus close-ended questions
It is certainly true that close-ended questions (whether numeric or multiple choice) are faster to answer than open-ended questions (where the respondent can say in their own words whatever they want).
It’s also generally true that open-ended questions require more work to analyze – and the extra effort required on the part of the respondent means that open-ended questions are more likely to be left unanswered.
Even so, sometimes you really need an open-ended question. In deciding when you should forego the easier closed-ended question and go for the added work of open-ended, there are three things to consider:
1 – when you don’t know enough about the topic to design a good multiple choice question
If you’re a Londoner designing a question about what Londoners like to do on a Saturday night, you’re probably well-equipped to come up with a list of multiple choice options that cover most respondents’ activities.
On the other hand, if you’re a Londoner designing a question about what Amazon tribespeople like to do on a Saturday night, you may have little or no idea of what the most popular activities are. You’ll need to do something to learn more before you can develop your multiple choice question. As noted by the Pew Research Center (which knows a little something about data collection):
Researchers will sometimes conduct a pilot study using open-ended questions to discover which answers are most common. They will then develop closed-ended questions that include the most common responses as answer choices. In this way, the questions may better reflect what the public is thinking or how they view a particular issue.
Other options besides conducting a pilot study (also called a pre-survey) for learning more could include:
- spend time speaking in detail with people similar to your potential respondent pool
- speak with colleagues who have more knowledge about the topic than you do
Note that each of these options adds time and effort (and probably money) to your effort. But the bottom line is that you can only design good multiple choice closed-ended questions if you have good knowledge of the topic. If you don’t have that, you’ll have to find a way to get it.
2 – when you have a lot of time and money
As noted above, open-ended data takes more time to collect, and more time to analyze. If you don’t have that time, or the money to pay other people for their time, open-ended is not going to work for you. If you have a lot of resources at your disposal, then all else being equal you may want to ask more open-ended questions.
3 – when you need a lot of detail
Open-ended questions may collect a greater level of detail than closed-ended ones, and you need to decide if that extra detail is needed, or worth the extra effort.
For example, if you’re interested in finding out whether people in a particular city generally use their cars for entertainment purposes, you definitely would not want to ask them an open-ended question like “what activity do you like to do on a Saturday night, and do you usually drive to that activity?” That is asking for more detail than you need. You could just ask a multiple choice question:
When going out for entertainment on a Saturday night, do you usually drive? Yes or No
That question only asks the level of detail required by your interest. It’s faster for the respondent, more likely to be answered (because it’s simple) and easier to analyze.
The internet has lots of great survey design information. Some of our favorites:
Also published on Medium.
Study: Improving Performance of Polio Programs
Systematic accountability frameworks are central to improving program performance through close monitoring of selected activities. The World Health Organization monitored the Nigeria polio program using Magpi, and used the resulting real-time data to drive positive changes in staff and outcomes.Download the Study