Getting Started with Early-Design Tests

So, how do we get started with early-design tests? For tree testing *and* for first-click testing,  we need to know what it is you're trying to find out. So, in the case of tree testing it  is focused entirely on the *navigation*, so trying to work out whether you've got the right  navigation terms: Are these words that people are understanding in the context of the problem?

So, if you're using odd words or words that are specific to your product line – for example, brand terms – then you may find that other people just don't understand them. And tree testing is one way to find that out. Apart from the terms themselves, how are they organized? Is your structure effective? Do people relate to it? Can they navigate through the structure? That's what tree testing will do. It shows them each of the menu structures one level at a time. And so, they go down a particular path; and if they get lost, they have to go back up that path,

and we record that information, and we call that a *qualified success*. If they go straight to it, it's *entirely successful*. If they have to bounce around a bit, then that is measured as successful but what's called *indirect success*. And then – as I already suggested – the big question of whether users can achieve important goals with your solution. So, are they actually able to do the things that you want them to be able to do? So, if you're an e-commerce site – to get through the checkout process or to

at least get to the products that they're interested  in, which is another big question mark area in many e-commerce solutions. If you're perhaps a marketing  site or some kind of recruitment site, then are people actually finding the things that you  need them to find in order to convert them – to get them to sign up to your project or to get  them to sign up to your newsletter – that kind of problem? I was visiting a charity site just the other day, and they'd written to me in the post. I'd actually had a piece of mail that I'd put into the  recycling because I thought, "Oh, I'll do that online."

Could I do it online? Heck, no! I couldn't. I couldn't find anything about the thing they'd just written to me on their website; so, – you know – if somebody had given me that goal: Find out about this particular type of contribution, I would have failed  miserably with the site as it was presented to me. And, of course, who do you need to target, in both of these cases? Are you talking about existing users? Are you talking about people who  perhaps have not seen the solution before at all?

(so, new or potential users) Or are you talking about a very specific segment of your users who need *particular features* – particular things to work in particular ways? So, that of course has a really big impact on  how you do recruitment and how time-consuming or potentially expensive recruitment becomes. I know from trying to recruit people, say, in finance or in legal professions that recruitment can get  extremely expensive when you start looking at

professionals in certain fields and with very  focused needs in terms of your activities. Response or respondent services are the easiest  way to get hold of recruitments for all of these online research activities. They recruit people to panels; and if you wish, you can actually join some of these panels. Just type in some relevant words  and add a "Join" or "Take part in" or "Participate in", etc. And you'll find some that will have  you just for the chance of winning some prizes

and others who will actually pay you per  questionnaire completed or per study completed. So, that's the kind of scenario that you're most  likely to find yourself in, but there may be other situations where respondent or response  services are not necessarily the best approach. If you're using some of the big names in user  experience, you may find that the tool that you're using does what you want but they don't offer  recruitment – a little bit unlikely perhaps, but it may happen – and in which case you could simply  go out to a recruitment agency,

somebody who will get you the participants that you need – not out of the goodness of their hearts, I hasten to add; this will all be for *money*. And how much money is  to a large extent determined by how fussy you need to be, how many people you're going to reject or  how complicated the recruitment questionnaire is. If you need *existing users*, then you may find that  you may already have a mailing list of people who might be willing to help you. If you're doing this on behalf of a customer of yours, then you may find that they have a mailing list.

I certainly have had experience of using customer mailing lists before now, and typically you get better response rates because the people on the lists have some kind of *affinity* with the organization whose  solution you're trying to improve. So, in many cases that can be extremely helpful. Otherwise, you can see if you can run an advert on the site itself to say that you're looking for research help and  would people be willing to spend 10-15-20 minutes

– you need to be accurate about that, but that kind  of area – in performing some research studies for you. Along those lines, do be prepared to offer  an *incentive* – you will get *very* few respondents purely out of the goodness of their heart. But  as soon as you start doing things like offering well-known brand gift vouchers, then interest  perks up straight away! But you do really need to try to keep the incentive in line with the amount of effort required on both ends of that graph. You don't want to be offering too little incentive –  people won't sign up.

And you don't want to be offering too much incentive, because people will be there primarily to get the incentive and will not necessarily be that engaged; so, it's really quite important to try to get the balance of that right. And then, I'll just repeat this because it is worth  repeating: When recruiting, do not understate the amount of time your studies will take to complete. And you can be caught by surprise on this if you've not done enough preparation. Some of the things that you'll find that happen is that you think that you've got maybe 10 minutes worth of

research in this particular project, and it only takes you or your colleagues 10 or 12 minutes to do it, but you get hold of a prospective user and they're entirely *dumbfounded* by your navigation or your tasks, and it's taking them 20-30 minutes; so, you do need to actually test these things out with prospective users. How many participants? Well, this is a  very difficult question at the best of times

for almost any kind of research, but it  becomes more complicated for these kinds of early-testing studies because the complexity of the tasks, the complexity of, in the case of tree testing, the navigation and, in the case of first-click testing, the complexity of the visual designs that you're using can have a really big impact on how good the data is; so, my recommendation here is that you try starting with  30 participants and then review the results.

But, having said that, start with 30 and perhaps repeat  if you're not seeing clear patterns as yet. Optimal Workshop recommends that you start at 50, which may be a simpler thing to do, but you would still want to make sure that you're happy with the quality  of the results that you're getting and they are significant. So, that we'll be looking at with a  chi-square test, which is a complicated name. Statisticians love Greek and Greek letters, so  chi is just a Greek letter.

And it's: you have to square the results – so, "chi-square". And we typically are looking for a confidence level of about 95%. That means that there is a  5% chance of you getting those results by *random chance*; and that's generally  taken to be adequate for most purposes. And if you're doing this in tranches – like I've  suggested, starting with 30, maybe adding another 30 and maybe adding another 30 – recruit more  participants incrementally and then go and look

at your results and see if you are still lacking  significance in some areas. If there are some insignificant results, don't worry – you may not be that interested, particularly in those specific tasks; they might have just been add-ons. But if the important tasks that you're setting your participants are still fairly vague, you will need to keep increasing the participant counts until you're happy that you've got something significant, which may not happen if it is a complicated task that users are not understanding

or if the navigation structure is not very good for supporting that task. You may discover that no number of participants is going to improve the results. So, by the time you reach 100 and you still don't have significant results on some of your tasks, you might want just to go back to the drawing board and try to find out where things are going wrong. And, like I said, certainly with the tree testing, you're going to get good feedback about what people were trying to do. Most testing tools allow you to define more tasks than you present.

So, you could have a bank of 50 tasks that you've agreed with your team or with your customer but decide that  you only want to present 10 to each participant; so, there would be a random selection of 10 that  each participant would be asked to go through. And 10 is generally recommended by most of the tools. It probably would take them 10 minutes in the case of both tree testing and first-click testing, but  that can vary – as I've said – depending on the overall complexity of the tasks and the thing  you're trying to test.

But if you are going to define, say, 30 tasks but only present 10 to participants, you're going to need 3 times as many participants because you have reduced the number of participants on each item by a factor of 3. So, that's the thing that you need to be  concerned with. Similarly, if you have 50 tasks but you're only showing 10 to participants randomly, then you'll need a factor of 5 increase in your participants; so, you're going to need to bear  that in mind if you want equal coverage.

And you will have no control over the coverage, by  the way; it will be just randomly selected – the subset that is presented, so bear that in mind when you're recruiting. For first-click testing, that is generally a bit quicker and a little bit less stressful than tree testing. Tree testing, as I say, you actually have to go through various stages,  one step for each level in the tree; so – and if you get lost, you have to go back, etc. With first-click testing, you're asking users where they would click to do something, and they click.

And that is the end of the story. There is no more to do. So, it is pretty quick by comparison, but you still wouldn't want to actually overload participants too much. *Setting user tasks* – that is something that's  essential to both of these early-design testing tools, and they need to be meaningful to potential users, and they need to be important for your solution.

So, if we're helping a customer to  redesign their website or redesign a mobile app, we would ask them to list for us tasks that are  important to them in the overall scheme of things. And you need to in the question *explain the  context* because it's very easy to forget that these things – the navigation, in particular, for the tree testing is being shown entirely out of context; there's nothing else on the page that gives people any clue about what this solution

is meant to do; so, they will have no inkling  that it's a clothing website or a political recruitment site or anything like that. So, you have to be careful to make sure that the tasks make sense in terms of the overall context. So, you might write things like "you're looking for a new business suit" or "you want to investigate a savings account for financial applications" or "you've received a confusing broadband bill" – so, you're talking about telecoms providers, that kind of thing. Do use *simple words and descriptions*. Keep things *short* but, of course,

describe the tasks adequately. But when you're doing that, don't give hints; don't ask leading questions, as it were. Make sure that you're asking it in a fairly natural way that you would hear people talking about in conversation. Do not use words that are part of your navigation structure, because you're trying to work out how people can  map what they're thinking of in their heads to actions that they can perform in your interface,  and that's via the navigation terms that you've selected.

And I'll give you a concrete example  because this is one of the situations where depending on how you ask the question, you would  expect to see totally different results because of the way these things are typically designed. So, imagine that you're talking about changing the expiry date on a credit card. In many cases I've seen, you actually have to know as a user that you've got to go through to your Account section,  you've got to find your Payment methods, you've got to *find* the credit card

and then you've got to actually work out how to change the expiry date. And it hardly ever says anything about changing  the expiry date; it usually ends up in many websites, for example, that you have to delete the old card and create a new one. Although changing expiry dates is something that most people have  to do at least once a year because it's necessary typically to do it every couple of years for  every card you have; and if you've got 3 or 4 cards for different purposes, for example, then  you might end up having to do this once a year.

And having to re-enter credit card details once a year is really not a particularly attractive thing to be doing. So, instead of asking users in the task that you're writing to edit their credit card details to change the expiry date – which is giving them a bit of a clue of how to do it – just *give them a scenario* – tell them that the card  has expired and they need to update the website, and see how they do, because in the first case, where  you're kind of telling them what they need to do, which is to edit their credit card  details, you would get a very much higher success rate

than the real-life situation of me finding  out that my credit card has expired and me even being told by the website that my credit  card's expired, and then having to work out how to address that. Websites that actually address that problem directly are very much more likely to be successful in terms of getting this information in  a timely fashion than those who just leave it as just a piece of data that users have to somehow manipulate.

And, like I say, in quite a few cases websites make you delete old cards and re-enter them completely. Amazon, to my delight, has been over the years one of the very few people to have got this right – that if you go to your Amazon payment cards and you look at a listing of them, the expiry date is a field that you can just easily change if you wish, without any real problems; so, their solution is quite a nice one.