User Interviews

As an interviewer, I think it's important to go in with an attitude of *openness*, with an attitude of *wanting to learn*, of *respecting the expertise* of your interviewee, whatever kind of expertise that is, and wanting to learn from them. I sometimes perhaps take that to extremes.

But I think that an important part of interviewing well is to listen to people and not to come with preconceptions about what they're going to say or what they can share. And the other, of course, is about an interviewing style that's open and respectful and *not aggressive* to people so that they do feel relaxed and able to say things and not feel judged for what they're sharing and what they're saying.

So, quite a lot of my research now is around health technologies, particularly. So, that can lead into some fairly sensitive topics. We've done projects looking at how people might use technology to manage their alcohol consumption or their exercise or their diet. And these are all topics that people can feel very defensive about. You know – "How dare you suggest I drink too much or I'm a couch potato or I eat too many carbs!" or whatever.

You know, they're topics where people *can* feel judged. But if people feel judged, then they're not going to give you a real understanding of how you might design technologies that will *really* help them. And so, it's really important to *not* be judgmental – to be open, to respect where they're coming from

and to understand what matters to *them*, as opposed to what you think should matter to them. Well, I certainly work quite hard with students to try to get them to question their own assumptions and not to expose their assumptions when they're interviewing – so that they are actually open to hearing what people are really saying, what people are really trying to express.

Another point is if interviewers are not *intentionally* leading. By coming with too many assumptions about what you're expecting people to care about or expecting people to say, you can unwittingly lead people quite a long way. And leading questions will result in you hearing what you expected to hear, which doesn't necessarily give you the information that you actually need to gain insight from any study.

And surely the point of doing interviews is to get some insight that you didn't already have. If you already knew the answer, there'd be no point in interviewing people about the topic, whatever it is. D.H.: You always have assumptions, right? Otherwise, you wouldn't do a study. A.B.: Yes. D.H.: So, what's the best way to sort of balance or use your assumptions in a constructive way? A.B.: So, I think what you try to do is to have questions more than assumptions.

A *qualitative* study I think is driven by a *question*. *Quantitative* studies are more often driven by a *hypothesis* – i.e. you have a belief and you want to test it – whereas qualitative studies are much more driven by questions. And I've certainly got partway through several studies and suddenly realized that I had a set of assumptions

that aren't necessarily valid. And so, I'm forced to question my own assumptions.

In a sense, the answer is in the expression  'semi-structured'. So, a completely structured interview is a *conversation where all the  questions are pre-scripted* and very often the answers are *closed answers*, so like an option 1 from 5 or a closed question like 'Yes' / 'No'.

And if you're going to do that kind of interview, really it's very much like a *survey*, and you might almost do it – it might be better as a written survey rather than as an interview. At the other end of the spectrum, you just  have a *conversation, a chat with somebody*, which might or might not be on a particular topic or might range across a whole spread of topics and cover all sorts of material not  necessarily with any particular structure and not with a particularly obvious purpose to it.

*Semi-structured lie between these extremes*, i.e. there is some kind of structure to it; you typically have a *topic guide* or a *semi-structured interview script* that determines the kinds of topics that you're going through and an initial planned order with which you'll cover those topics. But it's also free form in that people are welcome to give long answers, like the answers I'm giving you now.

And you can elaborate on questions; so, if somebody says something particularly interesting that you hadn't anticipated, you might follow up on that. And indeed sometimes people answer the topic that you'd thought of as being the fifth topic – they might introduce that when you thought you were still talking about the second topic; and  so, it kind of *free-flows* to some degree and feels natural. But as an interviewer, you have an agenda;

you have a set of topics that you want to cover; you perhaps even have some detailed plans of how you're going to ask some of the questions so that you get through the topics that *you* want to cover even if participants maybe go off topic at times or – you know – divert into other things. So, that's roughly what a semi-structured interview is. It's a *conversation*, but it  has a set of *topics* that you want to cover

and is *organized* such that you're reasonably  confident that you will cover those topics, but perhaps in a way that is responding to the participant and their interests and to some extent their agenda as well, as well as that of the interviewer, but it's not a completely fixed structure where you're not allowed to deviate from what you've planned ahead of time. In HCI, I think they're the most common form of interview because

we typically have an agenda; we're concerned with finding out from users or from potential users about the requirements for future designs or about the way that people do their work so that we can design new technologies to support that work or that activity better. Or maybe there's already an existing technology that you're testing,

and so you want to know specific things about what works well, what doesn't, and so you have an agenda as an interviewer; it's not just a casual conversation. But you do want to know a set of things about that thing. But you also want to hear what the user has to say; you want to listen to them, especially when they say things that are *unexpected*,

as well as finding out the answers to questions that you've asked. So, as interviews go, they are – I would say – the most common form used in HCI.

Ditte Hvas Mortensen: Thinking about questions, there's a certain sequence to how you best do it. So, maybe you could start by saying something about what the best way is to start an interview. Ann Blandford: For sure, the best way to start any interview is with opening questions that set people at ease, that assure them of what kinds of topics are going to be covered,

that give them a sense of what will be done with the data, though maybe that will be even before the interview starts, as I think about it. But it's about *setting somebody at ease*, about *helping to build rapport with them*. Obviously, each of us as an interviewer has our own personal style. And also every *interviewee* has their own personal style. And so, no two interviews are actually the same as each other.

I can't imagine ever running two almost identical interviews because they do so much depend on the participants – both the interviewers and the interviewees involved. But it is initially about setting somebody at ease, asking them comparatively innocuous questions – for example, what their role is in the organization if it's about a work system or their experiences of using a technology *like* the one that you're thinking about designing.

D.M.: So, it sounds like it's also pretty concrete questions. A.B.: Usually initial ones – it's best in my experience to make them reasonably concrete. One can move on to more abstract or more speculative questions later. Or, you know, questions that perhaps get at more sensitive feelings and values and emotions will come

later in a conversation when people have settled in and have started to feel comfortable in the situation, as opposed to, you know, starting with "So, how do you *feel about* _____?" – you know. That's not likely to set somebody at ease – if you kind of head straight into those things at the outset. And then, at the end, it's important to wrap up in a way that again leaves people feeling that they've said what

they want to say, that there aren't any topics that *they* thought were important as part of this interview, that they had an agenda, giving them a chance to articulate anything that you might have missed, and also giving them a sense of what's going to happen *afterwards*, you know: Are you going to give them a report back? Are you going to advise their managing director about new technology requirements?

What will be done with the data – what's the *value of the interview* for them? Some people really care about that; others perhaps less so. So, it's about being sensitive to what different people need. And in the middle – I mean, that's obviously the bulk of it – it's really about *planning it well ahead of time* so that you've made sure that you're covering all the topics that you're aiming to cover in

a rational and sensible order. As I've said already, people may sometimes answer and introduce topics that they've thought about already and answer future questions before you've even asked those questions. That requires you to be on your toes and think, "Oh yes, they've already answered that completely," or, "They've already *partly* answered that," and then, you know, picking up on what they've already said so that you're showing that you've been listening to them, and pursuing that topic a bit more later.

But it's about *making it flow as a natural conversation* as far as possible, while also *covering the topics that you want to cover*. So, it's important to get the structure so that it's a natural one that flows for most people – even if some people will run it differently. And part of checking that is about *piloting it* – you know – running through it with a friend or with somebody who at least should be able to *pretend* that they're a participant in that study first, to make sure that

you've got a coherent set of questions that are *comprehensible*, that are *using the user's language*, talking in terms that make sense to them and that they can engage with. So, you know, none of us gets interviews right the first time. It's usually worth trialing them out and going from there. D.M.: So, it sounds like you should try to follow the user's or the participant's

– if she brings something up earlier than you had expected, you just go along with it, or...? A.B.: That's my style, certainly, you know, because once somebody's in the flow if they're talking about things that you want to cover *anyway*, it just seems most natural, then, as a conversation to let them carry on on that line; and then, when they've finished, perhaps bring it back to make sure that you are covering everything that you wanted to cover

– because otherwise it starts to feel very disjointed and people may well forget the thing that they'd already half-started to say, and so you'll then have lost it forever. So, it's much easier if people can actually just carry on in the flow, as long as they're not going *wildly off-topic* for too long a time. And actually respecting participants can very often involve them going off-topic for some parts of it.

And you're gently trying to bring them back, and exactly how you do that probably depends on your interviewing style, actually. I personally probably let people run on a bit longer than perhaps some other interviewers would because I want to find an opportune moment to get people back on track. I think probably the worst one I had was – again, it was a little while back –

where somebody wanted to express a lot about the unions in their organization and was determined to tell me about industrial relations even though my focus was on *technology design*. They were seeing the introduction of new technology as being closely linked to other aspects of their relationship with management. And, of course, those weren't directly relevant to me.

But if I had just kept shutting them off, then I don't think they'd have talked properly about their attitudes to the technology, either; so, it was about respecting the other things that they felt that they wanted to say that were in *their* minds related, even though they weren't as directly relevant to the user interaction design – to me. So, they were less directly relevant to me. It might have taken me slightly longer to get all the information that I needed in that situation, than it would have done had they completely stuck to my script.

But on the other hand, it helped me to build rapport. I think it gave me better information. It certainly meant that I had a better relationship with the people I was interviewing in that setting.

Ditte Hvas Mortensen: Are there some common pitfalls that you should be aware of when your search for themes? Ann Blandford: *Confirmation bias* is, I think, the biggest one: that you think you've spotted something fairly early on in the data, which, particularly if that's something that supports a pre-existing assumption that you had – you know – before starting the study.

And so, it's so much easier to spot data that supports that position than it is to spot data that contradicts it, particularly if the contradiction is a little bit kind of *tangential*. So, that's why I kind of emphasize that you have to look explicitly for contradictory evidence in the data because that can be harder to spot. So, you have to ask yourself specifically "What would contradictory evidence look like?"

or "What *might* it look like?" You know – "Is there any of that? *Why* might it be there?" So, I think confirmation bias is the biggest. Another trap is there's one participant with slightly outlandish but very engaging and exciting views,

and they express themselves in a wonderful way, and you almost start writing the narrative as if that person is the central figure – as opposed to being a slight outlier in the data set. And so, their perceptions and their attitudes and values are given more *weight* than those of perhaps less articulate or less extreme participants. D.M.: Are there any other sort of common problems that you often experience students have when they do analysis?

And do you have some good advice for how to tackle those problems? A.B.: I think the big one is the one I've already mentioned – about getting overwhelmed by the data and not really knowing where to start or when to stop; having so many themes and not being prepared to let go of any of them;

or having difficulty working out which ones matter and which ones don't. And *data gathering* for some people is more exciting than data analysis. In practice, a good analysis takes longer than data gathering.

It takes longer to analyze 10 hours of interview data than it does to gather 10 hours of interview data, *even* when you take into account the time it took to recruit 10 appropriate participants in the first place, which can be non-trivial. But analysis is much more of a solitary activity, whereas data gathering is inherently social. You know – you're interviewing people; you're learning things; it's fun and engaging.

So, some people can get quite bogged down in analysis, I think. And it feels like a huge task, and you need to break it down into *small steps* of – you know – "I'll deal with three pages of data today." – you know – or, "I'll do *this* step of coding." or, "I'll do this little bit of stuff." You've got to break it down into manageable chunks.

And also, just dive in and *do* some, actually, you know. The longer you leave it – and it's just this pile of transcriptions or whatever – you know, the harder it is to get started and the more overwhelming it can feel. So, I think *feeling overwhelmed* is probably the biggest challenge, and not being prepared to drop themes or not being able to quite see how themes fit together to tell a bigger story.

That's of significance for HCI research or for technology design. And going off on a weird tangent in the analysis, and forgetting why you were doing the study in the first place. So, with those, I think the best way to deal with it is to *talk to people about it* – you know – to get somebody else involved,

even in just talking through ideas with you.