Best Practices for Qualitative User Research

No matter how big or small your user research project is, if you're going to do user research, you want to make sure you do it right. You don’t want to waste your time doing research that does not truly represent your users’ opinions and behaviors. That's a waste of both your and your research participants’ time, and it’s bad for business. The first step in doing user research right is to understand best practices for the method you are using, and why best practice is, in fact, best practice. If you understand this, you can ensure that your own user research is truly relevant, you can evaluate the validity of other people’s research results, and you can explain the benefits of using a certain method to stakeholders. Here, we'll consider the differences between qualitative and quantitative research and explain what the best practices are when you do qualitative user research.

Quantitative vs. Qualitative User Research

Different user research methods have different guidelines for best practice. User research methods are often divided into quantitative and qualitative methods. In quantitative methods such as surveys, analytics, and formal experiments, the researcher seeks to measure aspects of users and user behavior in a way that can be quantified and used for statistical analysis. Best practice for quantitative methods is to collect objective measurements that are unbiased by the researcher’s hypotheses, presence, and personality. Getting a sample of users who can represent the entire user population in a statistically meaningful way often requires many research participants. Qualitative methods such as interviews, ethnographic field studies, and (to some degree) usability tests are often more exploratory and seek to get a more in-depth understanding of the individual users’ or user group’s experiences, motivations, and everyday lives. In qualitative methods, the researcher accepts that he/she is to some degree a co-creator of research results, so the results are not completely objective and reproducible. The results of qualitative research are usually not expressed numerically, but rather as themes or categories that have occurred during the research.

To illustrate the difference, let’s say you want to study a user group’s exercise habits. You can choose to study these using either quantitative or qualitative research methods. If you use a quantitative method, you could create an online survey and distribute it to a large number of participants. Participants must answer predefined questions about their exercise habits such as “How many hours per week do you exercise?”. If you do your job dutifully, the survey answers can be numerically summarized without bias from your own opinion or personal experience with exercise.

If you use a qualitative method, you could choose to do interviews with a limited number of participants, where you talk to the participants about when, where, and how they exercise. Because the interview is similar to a conversation, your results will depend on how you ask follow-up questions to the participants’ answers – and how you do that will to some extent depend on your personality and your own experience with exercise. Similarly, the results of the interview also, to some extent, depend on a subjective interpretation of what the participant has told you.

Here, we'll focus on how to do qualitative user research of a high quality. Thus, we'll only bring up quantitative user research methods to highlight the differences between the types of research; however, before we go any further, we should mention that combining the methods often makes sense. For instance, many usability tests combine quantitative measures of (e.g.) time and errors with qualitative methods such as observations and interviews. Sometimes, researchers use the terms ‘objective’ and ‘subjective’ to describe the data collected through quantitative and qualitative methods. Quite often, in a usability test, you might combine both types of methods in order to interpret results. The term ‘objective’ refers to indisputable observations (facts) made during an experiment.

“A painstaking course in qualitative and quantitative analysis by John Wing gave me an appreciation of the need for, and beauty of, accurate measurement.”
— Paul D. Boyer, American Nobel Prize-winning biochemist, analytical chemist, and professor of chemistry

Let’s picture a scenario to illustrate this—imagine you're testing the interface of a mobile app for the task of uploading a new photo and participants completed the task on average in 5.5 seconds and clicking on the various control options 6.8 times. The term ‘subjective’ refers to the personal opinions and sentiments of the participants, which are individual to them; for example, 90% of users perhaps felt that the clicks required to complete the task were “too many” (we see these words often come up, in fact). To get meaningful results, you need both types of data. Although 5.5 seconds appears to be a short amount of time to complete a task, if the 6.8 clicks required for completing it are “too many” according to your participants, then clearly you need to redesign your interface.

So, quantitative research methods seek to measure an objective reality that exists externally from the researcher, and we have clear guidelines showing how to ensure that the research truly is objective and valid. With qualitative research methods, you cannot measure a completely objective reality that is not in any way influenced by the researcher. Instead, you work with the research participants to explore what the world looks like from their perspectives, or you interpret your own observations of each participant’s reality. This means that it can seem unclear how to perform qualitative research that truly represents a user’s perspective rather than your own biased understanding of his or her perspective.. This can lead to two problematic misunderstandings. One is that qualitative research is of less value than quantitative research, and the other is that qualitative research does not need to be methodical and structured, because observing or talking to people does not require any real preparation or knowledge (which then confirms misunderstanding number one!). Let’s look at each misunderstanding in turn.

Is Qualitative User Research of Less Value Than Quantitative Research?

Sometimes, we think of qualitative user research as something we choose to do when we lack the time and resources to do proper quantitative research. While time and resources certainly play a factor when planning user research, qualitative research has merit in its own right.

Qualitative methods are good when you don’t have a lot of information about the topic you want to research, because they allow you to be explorative. You don’t have to know in advance exactly what questions make the most sense for your research to ask, because you can keep an open mind and change your focus based on what the participant does or says. If we think of the exercise example, a quantitative survey requires that you know enough about your participants and how they exercise to know what questions are the most relevant to ask in your questionnaire. For instance, it only makes sense to ask a participant how often he uses a fitness tracker while exercising if he does—in fact—exercise from time to time and knows what a fitness tracker is. If, on the other hand, you were doing an interview, you could change the direction of your questions if you discovered that the participant does not know what a fitness tracker is.

Qualitative methods can give you an in-depth understanding of something in ways that are impossible to reduce to numbers. Qualitative methods can provide you with more detail about a situation or an answer because you can always follow up with new questions or go back to your participants for more detail. So, if your participants do not use fitness trackers, an interview allows you to ask them “Why not?” in a way that is open to all kinds of answers, whereas a survey requires you to list a limited number of possible reasons.

Finally, qualitative research methods do have the advantage in that they often require less time and fewer resources than quantitative research methods do; so, they're easier to fit into—e.g.—an iterative development process.

Does Qualitative User Research Have to be Methodical and Structured?

“The worst thing that contemporary qualitative research can imply is that, in this postmodern age, anything goes. The trick is to produce intelligent, disciplined work on the very edge of the abyss.”
—David Silverman, American professor and author of several books on qualitative research

We stated earlier that quantitative research seeks to collect objective measurements about an external reality without any influence or bias from the researcher and that qualitative research is not objective in the same way, because the researcher’s personality and experience will always, to some extent, influence the results. However, qualitative research should still strive for objectivity, although in a different way from quantitative research. In his classic book Interviews: An Introduction to Qualitative Research Interviewing, professor of psychology Steinar Kvale proposes three ways in which qualitative research can strive for objectivity:

• Through freedom from bias

• Through intersubjective knowledge

• Through being adequate to the object investigated

Each of these tells us something about how we should do qualitative research so that it is methodical and structured. With that in mind, let’s look at them in more detail.

1.Objectivity Through Freedom From Bias

When we do qualitative research, we should strive to be open and unbiased, so our research represents the participants’ perspectives as much as possible. Since no one can start a user research project without any kind of expectations or previous knowledge about the topic of investigation, we need to be consciously aware of any expectations and bias that we have when we do start one; if we aren’t, our expectations and bias might end up controlling us. So to carry out unbiased research, it's likewise vital to be systematic when you analyze interviews—to ensure that you truly explore our results and don’t just search for the results that you expected to get at the beginning of the project.

In the exercise interview example, we could have an expectation that people would be more motivated to exercise if they had an exercise partner, and we might even already have a good idea for an app that could help people find an exercise partner. While good ideas are great, it’s important that we still do the interviews with an open mind about how people feel about exercise partners. It’s important that we still ask open and explorative questions and that we don’t put words in the participant’s mouth by asking leading questions. Similarly, it is important that we don’t simply search for places where participants have stated that they would like an exercise partner when we analyze the results, but that we are open and explorative about what has actually been said. If we are not open to the participant’s point of view, there is no reason to carry out the user research.

Using an extreme example, but one that shows up just how glaringly bad this practice can get, imagine yourself being asked questions by a researcher who is adamant that the world is flat (and pretend you hadn’t been ready to rebuff his view when you entered the interview room). Picture his glee, for instance, when you admit that the sidewalk outside your home is flat, that every pool table you’ve ever played on was perfectly even, and that you’ve never noticed any substantial curvature when looking at the horizon from the ground. At the same time, imagine how he stops writing and frowns when you mention that you have, however, noticed the earth’s curvature from an airplane window... or how he asks you what your eyesight is like when you try to argue the point that everything beyond the horizon isn’t visible, due to the earth’s curvature. That researcher isn’t doing qualitative research—he’s out to confirm his beliefs by selectively picking at what you’re telling him, often lifting it out of context. That’s not really all that scientific, is it?

You can encounter the same issues when you do usability tests. If you think you already know where the problems with the interface are, it’s important to keep an open mind so that you don’t overlook problems that you did not expect or accidentally lead the user to find the same problems as you.

2.Objectivity Through Intersubjective Knowledge

Objectivity through intersubjective knowledge refers to an understanding of objectivity as something that multiple people agree on. In qualitative user research, that means ensuring that you are not the only person who interprets your data the way you do. One way of doing this is to have another person who might have a different perspective help you analyze your data. That way, you can compare your observations to see if you agree. You can also choose to share and discuss your observations openly with your team to see if they have a different interpretation of your research. Doing this requires that everyone be prepared to be open and curious about the data. Finally, you can ensure that your research participants agree with your observations and interpretations by continually sharing them with them. In the exercise interview, you could follow up answers that might be interpreted in different ways with a quick summary of what you believe the participant has said – e.g., “You said that you find it difficult to exercise in the evening and that you would rather watch TV, so is it correctly understood that you think evenings are for more relaxing activities, or is it something else?”. You could also choose to share your analysis or report with the participants to see if they have any comments or objections to your findings. Objections do not always mean that your interpretation is wrong, but they add to your understanding of your participants nonetheless.

3.Objectivity Through Being Adequate to the Object Investigated

Research is adequate to the object it investigates when the results show the real nature of the object. Both working in an unbiased way and discussing your interpretations with other people help ensure that your research reflects the nature of the topic you are investigating, but it is also important that you choose the research method that best fits your research topic. Research methods have different attributes and are good for different things, so what results you can get depend on what research method you choose. In the exercise example, the interview reflects the participants’ memories and opinions about how they exercise, but a form of observation of their exercise might reveal a truer picture of how they exercise. After all, we humans have this tendency to ‘misremember’ things, be it because we’re so used to an activity that we stop being mindful of what we do in it, or because we like to picture ourselves as being that bit more adept than we are, or because we might be prone to self-deprecation and humility dilutes the scale of our expertise.

Similarly, we can use interviews to talk to participants about the usability of a user interface, but observing how they interact with the user interface during a usability test (or a combination of test and interview) will probably give a more accurate picture of what they find difficult.

The Take Away

Here, we have focused on the positive attributes of qualitative user research and what it means to be objective in qualitative user research. We can sum up how to be objective when engaging in such research with the following best practices:

• Be open and unbiased – both when you meet research participants and when you analyze data from your research.

• Be aware of your own preconceptions.

• Discuss your interpretation of data with others, or have others help with the data analysis.

• Confirm your interpretations with your research participants – either during or following research.

• Choose the most appropriate method for the topic you are investigating.

These are best practices. They don’t represent what everyone does when doing qualitative user research, and sometimes circumstances are such that you cannot live up to all of them. But even when you can’t, you should know what to strive for so you know the limitations of your research. That way, you will enable a more scientific transfer of facts, ground your research in firmer reality, and vastly increase the likelihood that your projects end up engaging with real-world users in the way you hope.

References & Where to Learn More

Course: User Research – Methods and Best Practices

Steinar Kvale (1996) InterViews: An Introduction to Qualitative Research Interviewing. SAGE Publications.

Ann Blandford. “Semi-structured qualitative studies.” https://www.interaction-design.org/literature/book/the-encyclopedia-of-human-computer-interaction-2nd-ed/semi-structured-qualitative-studies?

You can read more about the strengths and weaknesses of qualitative research here.

If you are interested in learning more about quantitative and qualitative research, you can read:
John W. Creswell (2014) Research Design: Qualitative, Quantitative, and Mixed Methods Approaches, 4th Edition. SAGE Publications

Image

Hero Image: © Dave, CC BY-NC-ND 2.0