Usability Evaluation

If you just focus on the evaluation activity typically with usability testing, you're actually doing *nothing* to improve the usability of your process. You are still creating bad designs. And just filtering them out is going to be fantastically wasteful in terms of the amount of effort. So, you know, if you think about it as a production line, we have that manufacturing analogy and talk about screws. If you decide that your products aren't really good enough

for whatever reason – they're not consistent or they break easily or any number of potential problems – and all you do to *improve* the quality of your product is to up the quality checking at the end of the assembly line, then guess what? You just end up with a lot of waste because you're still producing a large number of faulty screws. And if you do nothing to improve the actual process in the manufacturing of the screws, then just tightening the evaluation process

– raising the hurdle, effectively – is really not the way to go. Usability evaluations are a *very* important tool. Usability testing, in particular, is a very important tool in our toolbox. But really it cannot be the only one.

We're going to be looking at a couple  of testing tools or testing techniques, one called tree testing and the other called first-click testing, which are *admirably suitable* for early design. So, you're working on – you're thinking of how your website should be navigated or you're working on some very rough wireframes or  general ideas about how that navigation should be

presented in the context of your project, and you'd like to get some feedback. And these two techniques are really absolutely ideal for that scenario. But, having said that, both of these techniques: tree testing and first-click testing can be used at  *any stage* in project development, so all the way from early design through to final buffing and  polishing, say, of the navigation framework. If you want to just make a tiny change and see  how that affects the site *before* you launch,

then something like tree testing may be absolutely ideal. And one of the real strengths of these tools is that they're pretty  isolated in the sense that you're only testing one or two very specific components of your overall solution. So, tree testing in fact is absolutely laser-focused on your navigation; that is all you're presenting, and you're not presenting it with any kind of visual component.

It's presented, it's simulated by the testing tool. So, you just upload a navigation tree that you want to test and you give your participants some goals and off they go; and you're testing now  how they understand those words – nothing to distract them; there are no visual distractions at all. But they do lose a little bit of the context that way, as we'll see a bit later on, too. So, early-design testing allows design components to be evaluated *prior* to extensive development efforts.

So, you could be doing this in the first week of a new project. There would be no problem at all in that. It might take a little while to *organize*, but certainly you could be seeing results only  a couple of weeks into your project. There are *many* forms of early-design testing. The particular attraction of these is that they are what we call *quantitative* – they produce numbers where we get to  actually see some statistics on how people perform in these particular tasks.

Paper prototyping I list as an alternative here. That is not a quantitative process *typically*. We're not actually looking mostly at success when we're talking about paper prototyping. We're looking at a *qualitative* focus, so we're trying to understand what's going through people's minds and we ask them about  why things didn't work and what's going on inside their head. That's very similar to usability testing, and usability testing is predominantly a qualitative process.

So, these are almost entirely quantitative; they're done *online*, and that means that we're not really going to be able to get  inside our participants' heads, but that is both a benefit and a drawback. The benefit is that we're going to get some hard numbers out of it and it's very quick to do. We will be typically testing with dozens, scores of participants – certainly, 100 participants wouldn't be difficult to imagine. Whereas in usability testing, a day's work is represented by about seven

participants, maybe up to 90-minute sessions, so they are quite different animals from that point of view. And I won't be talking any more about usability testing in this. It was just by way of comparison. And usability testing, of course, as I mentioned is a qualitative technique. The *reasons* that you might consider early-design testing – well, it's relatively quick and inexpensive; done online, primary costs with that are actually *recruitment costs*

– getting people to come and take our studies; so, there isn't the renting of a lab for any kind of qualitative research; there isn't the hiring of researchers; You set up the project; you leave it to run; you recruit participants to it, and you check on your results a few days later. It can be as straightforward as that. Only minimal navigation or visual design is needed; you don't have to have thought things through absolutely to the end of the process.

You can get some really good feedback about what's working and what it isn't from the very earlier stages, and again that's really one of the main strengths of the whole approach in both of these cases. Very effective for early design, and – as I mentioned  already – can be used at any stage in a project. And I've hinted at this, but I will now put it in very precise terms that it's a *goal-oriented focus*; in all these cases we present our navigation or our visual design along with some tasks

– a list of tasks that we would like users to try to pursue. And we're looking at how they *perform*; so, that is the quantitative aspect of it. It is purely *what they do*. There is no element of asking them how they feel about it or why they've made those particular decisions – although those are optional extras in some cases. Certainly, the *why* aspect of it can be; but *predominantly* we're talking about the numbers that come out of the end. And that's why we need at least several score participants

– something around the 30–50 mark as a starting point. For existing solutions, you could use early-design testing for identifying problem areas. So, if you've got some problems on a page or with a site in general but you just don't really have a handle on what those problems are, then trying to give people some specific  tasks and then giving them your navigation structure may produce some *really* very straightforward feedback on that issue.

You also can use it for collecting data for improving designs, along similar lines; you would try out things that you know people seem to be struggling with or that are really important to your organization's goals with a particular solution and see how people fare. *New solutions* – well, evaluating and improving design elements; it's really more or less exactly the same process, but you might be at the beginning of a new solution, be trying to understand what was good, what was bad about the *old solution*.

Certainly, knowing what worked is a good situation to be in when you're setting off to revamp a design. *Alternatives* – well, I already hinted at paper prototyping, and I've already said that that is primarily qualitative; we're talking about very small numbers of people for a fair bit of researcher effort. But it is different and it is qualitative, so we won't be going into detail about that here. Web app or analytics – the problem there is that we know what people do but we don't really know why they've done it.

So, we might find that people were bouncing between two pages in a website or a mobile app and not really understand *why*. So, we've got numbers that say "There's something going on here" but no way of finding out. So, if you have suspicions that it's navigation-related or the visual design we test with the first-click testing, then we can try that out pretty quickly and relatively cheaply, compared with some other approaches – particularly the qualitative ones, which do tend to be more expensive.

And, finally, card sorting is another technique you might consider. It can be used both qualitatively and quantitatively. And we won't be talking about that in these modules, but  you will find information on card sorting on the Interaction Design Foundation website. There's a very extensive encyclopedia article that I wrote about card sorting several years ago, which you should find pretty useful in that respect.

There is something called esthetic usability effect, which means that if something looks good, it's not really perceived by people as being more usable, even if it's not. So those visuals are important and we really, really need to keep that in mind. A lot of the users have now become very tech savvy. A lot of the people, you know, using apps and using websites have been using them for 15 or 20 years now, and they are very skilled.

So they understand the typical patterns. They know that checkout icons should be in the top right corner. They know how a good logging and registration works. So all things like that are kind of predetermined and there's pretty little innovation here. So you can do research to test the viability of the product. There isn't really much way to innovate and user interface. The visuals can be the differentiator. And of course we can use a design system like material design, but that would be

a horrible world to live in because those apps would be all looking boring and soulless and people want that design. That's why people buy Apple products, among other things, because they look good and they are different than everything else, at least sometimes. So people buy with their eyes and we really need to remember that.

One of the early standards that  mentioned usability was ISO 9241. And it talked about three crucial  issues for user interfaces. One of them was *effectiveness* – does it do the right thing? Does it get things done that are important? The second was *efficiency* – does it do that with the minimum effort? The minimum mental effort?

The minimum physical effort?  Or is it taking extraneous effort that's unnecessary? And very often, people only quote those two because there was a third one as well, which is *satisfaction*: Does it make you feel good? Do you feel happy having used this system or used this piece of software? And so, that last one is often missed entirely. And that's all about the *emotion*, the way you feel.

And it was often ignored, often missed in the  past. What's now happened is that's become perhaps in some ways more important than the other two. Emotion is important because it's good to feel emotion. But also, *emotion affects the bottom line in business*. If your employees are happy, they tend  to be more productive. So, if you're designing a production line or  an office or wherever the environment,  

if you can have software and systems that  make people feel good, they'll tend to work better. And certainly you want your customers to feel happy because they're the people who  are usually going to buy your goods. So, if you've not made your  customers happy, they don't buy anything. So, emotion is important to us as humans, but  it's also important from a business point of view.

So, semi-structured interviews – well, any  interview, semi-structured or not, gets at people's perceptions, their values, their experiences as they see it, their explanations about why they do the things that they do, why they hold the attitudes that they do. And so, they're really good at getting at  the *why* of what people do,

but not the *what* of what people do. That's much better addressed with *observations* or *combined methods* such as contextual inquiry  where you both observe people working and also interview them, perhaps in an interleaved way about why they're doing the things that they're doing or getting them to explain more about how things work and what they're trying to achieve.

So, what are they *not* good for? Well, they're not good for the kinds of questions where people have difficulty recalling or where people might have  some strong motivation for saying something that perhaps isn't accurate. I think of those two concerns, the first is probably the bigger in HCI

– that... where things are unremarkable, people are often *not aware* of what they do; they have a lot of *tacit knowledge*. If you ask somebody how long something took, what you'll get is their *subjective impression* of that, which probably bears very little relation to the actual time something took, for example. I certainly remember doing a set of interviews some years ago

where we were asking people about how they performed a task. And they told us that it was  like a three- or four-step task. And then, when we got them to show us how they did it, it actually had about 20, 25 steps to it. And the rest of the steps they just completely took for granted; you know – they were: 'Of course we do that! Of course we—' – you know – 'Of course that's the way it works! Of course we have to turn it on!' And they just took that so much for granted that *it would never have come out in an interview*.

I mean, I literally can't imagine the interview that would really have got that full task sequence. And there are lots of things that people do or things that they assume that the interviewer knows about, that they just won't say and won't  express at all. So, interviews are not good for those things; you really need to *observe* people to get that kind of data. So, it's good to be aware of what interviews are good for and also what they're less well-suited for. That's another good example of a kind of  question that people are really bad at answering,

not because they're intentionally deceiving usually, but because we're *not* very good at *anticipating what we might do in the future*, or indeed our *attitudes to future products*, unless you can give somebody a very faithful kind of mock-up

and help them to really  imagine the scenario in which they might use it. And then you might get slightly more reliable  information. But that's not information I would ever really rely on, which is why *anticipating future product design is such a challenge* and interviewing isn't the best way  of getting that information.

I wanted to say a bit more about this important issue of recruiting participants. The quality of the results hinges entirely on the quality of the participants. If you're asking participants to do things and they're not paying attention or they're simply skipping through as quickly as they can – which does happen – then you're going to be very disappointed with the results

and possibly simply have to write off the whole thing as an expensive waste of time. So, recruiting participants is a very important topic, but it's surprisingly difficult. Or, certainly, it can be. You have the idea that these people might want to help you improve your interactive solution – whatever it is; a website, an app, what have you – and lots of people *are* very motivated to do that. And you simply pay them a simple reward and everyone goes away quite happy.

But it's certainly true with *online research* that there are people who would simply take part in order to get the reward and do very little for it. And it comes as quite a shock, I'm afraid, if you're a trusting person, that this kind of thing happens. I was involved in a fairly good-sized study in the U.S. – a university, who I won't name – and we had as participants in a series of studies students, their parents and the staff of the university.

And, believe it or not, the students were the best behaved of the lot in terms of actually being conscientious in answering the questions or performing the tasks as required or as requested. Staff were possibly even the worst. And I think their attitude was "Well, you're already paying me, so why won't you just give me this extra money without me having to do much for it?" I really don't understand the background to that particular issue.

And the parents, I'm afraid, were not a great deal better. So, we had to throw away a fair amount of data. Now, when I say "a fair amount", throwing away 10% of your data is probably pretty extreme. Certainly, 5% you might want to plan for. But the kinds of things that these participants get up to – particularly if you're talking about online panels, and you'll often come across panels if you go to the tool provider, if you're using, say for example, a card-sorting tool

or a first-click test tool and they offer you respondents for a price each, then be aware that those respondents have signed up for this purpose, for the purpose of doing studies and getting some kind of reward. And some of them are a little bit what you might call on the cynical side. They do as little as possible. We've even on card sort studies had people log in, do nothing for half an hour and then log out and claim that they had done the study.

So, it can be as vexing as that, I'm afraid. So, the kinds of things that people get up to: They do the minimum necessary; that was the scenario I was just describing. They can answer questions in a survery without reading them. So, they would do what's called *straightlining*. Straightlining is where they are effectively just answering every question the same in a straight line down the page or down the screen. And they also could attempt to perform tasks without understanding them.

So, if you're doing a first-click test and you ask them, "Go and find this particular piece of apparel, where would you click first?", they'd just click. They're not reading it; they didn't really read the question. They're not looking at the design mockup being offered; they're just clicking, so as to get credit for doing this. Like I say, I don't want to paint all respondents with this rather black brush, but it's *some* people do this. And we just have to work out how to keep those people from polluting our results. So, the reward is sometimes the issue, that if you are too generous in the reward

that you're offering, you will attract the wrong kind of participant. Certainly I've seen that happen within organizations doing studies on intranets, where somebody decided to give away a rather expensive piece of equipment at the time: a DVD reader, which was – when this happened – quite a valuable thing to have. And the quality of the results plummetted. Happily, it was something where we could actually look at the quality of the results and

simply filter out those people who really hadn't been paying much attention to what they were supposed to be doing. So, like I say, you can expect for online studies to discard been 5 and 10% of your participants' results. You also – if you're doing face-to-face research – and you're trying to do quantitative sorts of numbers, say, you'd be having 20 or 30 participants, you probably won't have a figure quite as bad as that, but I still have seen, even in face-to-face card sorts, for example,

people literally didn't *understand* what they were supposed to be doing, or didn't get what they were supposed to be doing, and consequently their results were not terribly useful. So, you're not going to get away with 100% valuable participation, I'm afraid. And so, I'm going to call these people who aren't doing it, and some of them are not doing it because they don't understand, but the vast majority are not doing it because they don't want to spend the time or the effort; I'm going to call them *failing participants*. And the thing is, we actually need to be able to *find* them in the data and take them out.

You have to be careful how you select participants, how you filter them and how you actually measure the quality of their output, as it were. And one of the big sources of useful information are the actual tools that you are using. In an online survey, you can see how long people have spent, you can see how many questions they have answered. And, similarly, with first-click testing, you can see how many of the tasks they completed; you can see how long they spent doing it.

And with some of these, we actually can also see how successful they were. In both of the early-design testing methods – card sorting and first-click testing – we are allowed to nominate "correct" answers – which is, I keep using the term in double-quotes here because there are no actually correct answers in surveys, for example; so, I'm using "correct" in a particular way: "Correct" is what we think they should be doing when they're doing a card sort, *approximately*, or, in particular, when they're doing a *first-click test*,

that we think they ought to be clicking around about here. Surveys as a group are a completely different kettle of fish, as it were. There are really no correct answers when you start. You've got your list of research questions – things that you want to *know* – but what you need to do is to incorporate questions and answers in such a way that you can check that people are indeed *paying attention* and *answering consistently*. So, you might for example change the wording of a question and reintroduce it later on

to see if you get the same answer. The idea is to be able to get a score for each participant. And the score is your own score, about basically how much you trust them or maybe the *inverse* of how much you trust them. So, as the score goes up, your trust goes down. So, if these people keep doing inconsistent or confusing things, like replying to questions with answers that aren't actually real answers – you've made them up – or not answering two questions which are effectively the same the same way, etc.,

then you would get to a point where you'd say, "Well, I just don't trust this participant," and you would yank their data from your results. Happily, most of these tools do make it easy for you to yank individual results. So, we have to design the studies to *find* these failing participants. And, as I say, for some these tools – online tools we'll be using – that is relatively straightforward, but tedious. But with surveys, in particular, you are going to have to put quite a bit of effort into that kind of research.

Steps we can take in particular: Provide consistency checks between tasks or questions. Ensure that "straightlined" results – where people are always answering in the same place on each and every question down the page – ask the same question again in slightly different wording or with the answers in a different order. Now, I wouldn't go around changing the order of answers on a regular basis. You might have one part of the questionnaire where "good" is on the right and "bad" is on the left;

and you might decide to change it in a completely different part of the questionnaire and make it really obvious that you've changed it to those who are paying attention. But whatever it is that you do, what you're *trying* to do is to find people who really aren't paying much attention to the directions on the survey or whatever the research tool is, and catch them out and pull them out of your results. And of the issues you should be aware of if you're paying for participants from something

like your research tool *supplier* is that you can go back to them and say, "These people did not do a very good job of completing this survey, this study." And ask them to refund you for the cost of those. You tell them that you're having to pull their data out of your results. Also, it helps to tidy up their respondent pool. Perhaps it's not your particular concern, but if you do end up using them again, it would be nice to know that some of these people who are simply gaming the system have been removed from the respondent pool.

So, reporting them – getting them removed from the pool – is a sensible thing to be doing. And, finally, devising a scoring system to check the consistency and also checking for fake responses and people who are just not basically doing the research as you need them to do it.