Ditte Hvas Mortensen: Are there some common pitfalls that you should be aware of when your search for themes? Ann Blandford: *Confirmation bias* is, I think, the biggest one: that you think you've spotted something fairly early on in the data, which, particularly if that's something that supports a pre-existing assumption that you had – you know – before starting the study.
And so, it's so much easier to spot data that supports that position than it is to spot data that contradicts it, particularly if the contradiction is a little bit kind of *tangential*. So, that's why I kind of emphasize that you have to look explicitly for contradictory evidence in the data because that can be harder to spot. So, you have to ask yourself specifically "What would contradictory evidence look like?"
or "What *might* it look like?" You know – "Is there any of that? *Why* might it be there?" So, I think confirmation bias is the biggest. Another trap is there's one participant with slightly outlandish but very engaging and exciting views,
and they express themselves in a wonderful way, and you almost start writing the narrative as if that person is the central figure – as opposed to being a slight outlier in the data set. And so, their perceptions and their attitudes and values are given more *weight* than those of perhaps less articulate or less extreme participants. D.M.: Are there any other sort of common problems that you often experience students have when they do analysis?
And do you have some good advice for how to tackle those problems? A.B.: I think the big one is the one I've already mentioned – about getting overwhelmed by the data and not really knowing where to start or when to stop; having so many themes and not being prepared to let go of any of them;
or having difficulty working out which ones matter and which ones don't. And *data gathering* for some people is more exciting than data analysis. In practice, a good analysis takes longer than data gathering.
It takes longer to analyze 10 hours of interview data than it does to gather 10 hours of interview data, *even* when you take into account the time it took to recruit 10 appropriate participants in the first place, which can be non-trivial. But analysis is much more of a solitary activity, whereas data gathering is inherently social. You know – you're interviewing people; you're learning things; it's fun and engaging.
So, some people can get quite bogged down in analysis, I think. And it feels like a huge task, and you need to break it down into *small steps* of – you know – "I'll deal with three pages of data today." – you know – or, "I'll do *this* step of coding." or, "I'll do this little bit of stuff." You've got to break it down into manageable chunks.
And also, just dive in and *do* some, actually, you know. The longer you leave it – and it's just this pile of transcriptions or whatever – you know, the harder it is to get started and the more overwhelming it can feel. So, I think *feeling overwhelmed* is probably the biggest challenge, and not being prepared to drop themes or not being able to quite see how themes fit together to tell a bigger story.
That's of significance for HCI research or for technology design. And going off on a weird tangent in the analysis, and forgetting why you were doing the study in the first place. So, with those, I think the best way to deal with it is to *talk to people about it* – you know – to get somebody else involved,
even in just talking through ideas with you.