Writing Good Questions for Surveys

We're going to be talking  about writing good questions. The quality of the questions, along with the quality of the respondents, is really key. So, you have to have questions which people understand, that they really can address *unambiguously*; they don't have to sit wondering what it is you meant by that, and to do it pretty quickly, too – they need to be *short*.

So, we have a long list of various  points you should take into account. And it starts with the the issue of *shortness and ambiguity*. So, we want things to be as *short as possible*. We want to use short questions, short words. We want to be *very specific*, and we want to be *very unambiguous*. So, you don't ask vague questions. If you want to know about whether somebody has done something, you need to specify over what period. A lot of questionnaires these days I noticed ask,

'What did you do yesterday?' or 'Did you watch this yesterday? Did you do that yesterday?' And you could of course change that period as required, but it's very unrealistic to expect people to either mind-read if you're not being at all clear about when you mean, or to remember very far back. And that changes with the topic, of course. Somebody might remember when they were married from a long time ago, but the last time they visited a coffee shop – you might not get away with more than a couple to four weeks, for example.

Do use *common terms* and  provide *explanations* where you're using anything that's a little bit unusual. And common terms in  English are generally fairly short. So, use short words. Short words appear more frequently, are used more frequently. The number of syllables is actually a good indicator; the more syllables, the less frequent the word is used in English. And with a lot of interactive survey tools, you are able to provide explanations.

If nothing else works, then just put something parenthetically after the question to explain what it is you mean by that particular term. Start the questionnaire or the survey with *easy, uncontroversial questions*. If you're going to go on to be asking  sensitive questions or things that people might get slightly aggravated or upset about,  then try to leave that as late as possible. You want people to get engaged; you want them to feel comfortable, and to an extent trust you.

And starting with controversial questions or in some way irritating questions is not a good way to do that. Do *use words to label all points on a scale*. I've got a terrible question that I've made up about Brexit. But I've labeled every single point. Now, this used to be for me a debatable issue, that I suggested that perhaps we should just label  the two ends and number the points in between, but I've seen good evidence and firm advice that you should label all of the points in between the two.

Certainly on seven – this is a seven-point response – five or seven is fairly typical. And in those cases, you would expect to see words being used there. Address only *one aspect of a question at a time*. *Avoid compounds* like: 'I found the website quick and easy to use,' because there is the possibility that somebody found it easy to use but not quick, or quick but not easy. So, stop that from being a problem by asking those questions separately. Just like in navigation design, *do not use overlapping categories or ranges*.

People need to be very clear about where they should be clicking; so, you want '20-49', '50-64,' '65 and over', rather than '20-50', '50-65', '65 and over' because you can see in that latter example that there are two places  where people, if they were exactly 50 or exactly 65 would not be clear about where you actually want them to answer. And *do not ask leading questions* if you want honest responses,

or any responses at all, for that matter. Certainly, it's during various elections that have taken place over the last couple of years, I have seen numerous alleged surveys come round where clearly it is not a questionnaire or a survey at all; it is a party political statement, and I just stop  as soon as I realize that that is the case. And I believe a lot of other people do as well.  So, this example on 'How bad an idea is Brexit?' is an example of a leading or loaded question because of course some people think that Brexit is a perfectly good idea

and we should not be talking about it as a bad idea in their eyes. So, this is something that you would not do.  In fact, there are quite a few things in that particular example you should not do, so just  treat that as a bad example... *Avoid question grids*; I know that they  are extremely popular, but if you can avoid them, they are worth avoiding because  they are intimidating, certainly with the large question grids, people turn over to them online typically, and they immediately get put off. It looks very complicated. It is quite complicated.

If they were on the brink of not completing your questionnaire, that probably has pushed them over the edge. And if you're unlucky and are working with a service provider for your questionnaire, your survey tool, that doesn't support converting these into individual questions, then you'll find that they don't actually work on mobile phones. And that certainly does happen from time to time,  and of course we can expect more people to be

using mobile phones as their primary internet  tool; so, that would be a bad plan. Really the best way of approaching these is to ask the questions separately. And this is actually quite a bit more likely in the problem domain that we're talking about, which is user experience, rather than, say for example, market research where you might have a list of 10 or 20 coffee shops down the left-hand  side and the frequency of use across the top. That's still not easy for respondents, but it's  perhaps a little bit easier than asking them

deep, meaningful questions like this  particular one about Barack Obama. *Ranking questions* are those where you're asked  to re-order the responses. And the *problem* with them is twofold. One is that it's actually not very easy as a participant to do this, that you have to think about, 'Well, which of these is my preference? Which is my second preferred, third preferred?' etc. And it's off-putting to them; it's time-consuming; it's hard to do in some cases,

particularly on a mobile platform, but it might be  quite technically challenging, just the dexterity required, and it's also off-putting; you're making people think really hard about something where the detailed answers are not all that important.  What does it matter to you in a list of five whether somebody lists something  fourth or fifth? It's not first; it's not second; so, do you really care that  much about it?

It turns out to be very easy and almost equivalent. I have not seen a paper saying  that they're equivalent, by the way, but certainly when I've done it, I've not been disappointed. Just ask people to choose their *favorite*. Or, if you've got a long list, maybe their favorite  top 'n', where 'n' might be 2, 3, 4 ... And you're not talking about order, then; you're just talking about which is the most favorite or which are the two most favorite. And when you come to  analyze those, the analysis is very much simpler because it is just the items with the biggest  numbers are the most popular.

And that was the second point with the whole ranking thing – is that you have to do a *weighted analysis* in order to get sensible results from the ranking, which means  taking into account where in the list these things appeared; whereas if you're just asking people  to choose their favorite, it is very much easier and it's just slightly more complicated for their favorite two or their favorite three. And most survey tools will let you set a *maximum and minimum number of responses* for this kind of question.

So, if you really insist on having two choices from a  long list, then it could complain to participants that they've not selected two or preferably  a maximum is the better way of doing that so people cannot choose the full number if they  really don't care that much. So, that is something I would just strongly recommend you avoid altogether. It's really of no particular benefit. It looks fun when you're looking at it in the tool and  maybe on the screen yourself.

But when you're talking about lots of respondents getting to  it and dealing with it, it's not fun for them. Open-ended questions do allow some flexibility,  and of course they're not in any way going to replace an interviewer who can dig a lot more  deeply and try to interpret or understand what participants are saying. They have got their use, though, open-ended questions; the set of possible answers is unknown

or very large, is the main reason for doing that. So, if you've got a list of items and there are other possibilities, you will have an 'other' response, and under the 'other' response you will have a text box for people to fill in. Perhaps you want to know the underlying cause of a response – 'Why did you rate us like that?'; 'What was the main thing?' And that's certainly very commonly done and a perfectly good use of open-ended responses. Or you need to allow participants to express *unanticipated concerns*.

So, 'Is there anything that we could have done to have prevented this or to make you happier?' – that kind of question. And these two examples from SurveyMonkey: the top one is the final question in most SurveyMonkey templates. It's open-ended. And that's recommended for almost all surveys, that you give people a chance to comment about either your organization or the questionnaire or just general comments that they might have. 'Is there anything else you'd like to add?' is the kind of question you would ask there.

The bottom one is actually from their market research  template, and it has a lot of open-ended questions in it, but it's one of the very few SurveyMonkey templates that actually has more than one or two open-ended questions; so, you really can  largely – and perhaps should try largely – to stick to multiple choice questions. An alternative – and you may see this yourself as a respondent to various questionnaires – an alternative to extensive open-ended questions is to invite survey participants to take part

in an *online or telephone interview*. So, you might ask them a few questions, and they might express some reservations  about certain aspects of your product or service. And if they did that, you might offer them the  chance to be interviewed at depth, either by telephone or through some online collaboration tool. And of course that becomes, then, a primarily qualitative approach, and you would only do this with a relatively small number of participants,

mostly because it will be moderately time-consuming; you could expect it to take at least half an hour, and you will need a qualified  interviewer to do that. Do make sure that if you're using this approach that the interviewer does have access to the respondent's initial  questionnaire so they're not repeating themselves  or are not at all aware of the participant's background or complaint. *Semi-structured interviewing* is the most appropriate technique in most cases. So, this would be a follow-up set of questions and then just the interviewer exploring

some of the responses that the participant has given already.