Mental Processes

I'd like now to distinguish two words, which sometimes you'll probably hear me use interchangeably, but actually have a subtle difference: sensation and perception. In one sense, there's the things that we really hear, and shall we say, the immediate senses. This is what in psychology is called sensation.

So with your eye, the photons actually come into the back of your eye, hitting your retina and the nerve cells, noticing those and creating a signal. When you hear, little hairs in your ear are getting wiggled by the sound and then passing that into your brain. So that is the raw sensation. That's the absolute raw sense of what's there.

However, our brains do lots of work with that. They process what are those raw sensations in order to give that sense of that feeling of actually what's there. There's a couple of examples of this you can think of. So, for instance, if you... Looking at your room, just glance at the room you're in, you'll have a sense that you can see it all, but actually your eyes dance around and builds it up slowly.

So that raw sensation at any one moment is probably that everything in the periphery is fuzzy. But you don't see that. What you believe you see is something that's sharp everywhere. Because if you ever need to know about a bit of the room your eye will glance to it and it will become sharp at that moment. So your perception of the room and your visual perception is that it's large, it's everywhere, and you can see it all. Your actual sensation is much, much tighter than you can see clearly.

Because these differ and because they are about the meaning of the world... We are *meaning-giving creatures*; we're trying to find meaning and sense in this world. You can sometimes use this creatively to create effects, but you could also sometimes get, shall we say mistaken effects because of this. And optical illusions use this very much in order to fool your eyes because your eyes are trying to create sense of the world and you can sometimes fool them.

So here's two optical illusions, both of which are based on linear perspective. There's the Ponzo one. So if you look at that picture, it's a ladder with two blocks, and it will look as if the block at the top is bigger than the block at the bottom. The reason for that is your meaning bit of your brain is saying, "It's doing, the ladder's doing this," so it's probably sloping away from you. If it's sloping away from you, then something that takes the same amount of visual space

further away must be bigger. On the right is the Müller-Lyer illusion. So the Müller-Lyer illusion is two lat[eral] lines and then two arrowheads and one has got arrowheads going in and one's got arrowheads going out. And if you look at it, you probably will see the one with the arrowheads going out as being a longer line than the one at the bottom. In fact, they're absolutely the same size and the same length.

However, your eye sees the one at the top longer because it looks a bit like the back edge of a box – so, again, in terms of perspective, if the line's going... It's like looking at the back of the box whereas the ones at the front is more like if you're seeing the front end of the box. And of course, again, your eye is saying that probably means the one on top is further away. Although it takes the same amount of distance on the back of your eye, that probably means it's actually larger. Now all that's happened, when I say "probably", you're not reasoning this out.

This is all happening tacitly, unconsciously. You've got no idea it's going on. And it's very hard, even though you know those two are the same length to actually be able to see that. Now, it was thought for a long time that a lot of us... and some of these illusions really are very basic; they're built into our – shall we say – our most primitive base being, the things that we've had for, you know, 10-20 thousand years,

probably longer as part of the way we see the world. And it was thought in particular the Müller-Lyer illusion was one of these. Something that's very, very basic. I know I found problems with this for many years because linear perspective, it's actually hard to think, shall we say, what in the wild corresponds to linear perspective. And sure enough actually, what's been realized relatively recently is that most subjects of psychological experiments are what's been called WEIRD.

By "WEIRD", it's White, Educated, Industrialized, Rich and Democratic. So basically, think of US psychology students as subjects of most psychology experiments – or British ones for that matter. So you actually end up with a very select set of people

for which we believe everything is true of. And of course, actually that turns out not to be true. Crucially, an awful lot of things which appear to be very fundamental psychological effects turn out to be *cultural*. And crucially, this thing about the Müller-Lyer illusion is one of the ones that is not the same. So if you have a... there is a relatively small number of experiments where this is the case, but there are some experiments that have been done with people who have been brought up in

the middle of forest jungle environments, but without built straight lines, concrete and brick structures. And when that's the case, the Müller-Lyer illusion doesn't work. They see the two lines as the same length. This is a learned effect that's happened from childhood, where because you live in a built environment, you end up with particular illusions. So these are partly about sensory perception, but perception is based partly on

– shall we say – built-in parts of your brain. But also the culture in which you live changes the relationship between raw sensation and perception. So, you know, optical illusions can fool you. However, actually, if you think about it... So there's the kind of optical illusion like Müller-Lyer, but also you've probably seen those trick photographs where you think something's bigger than it is

or you see something and it says, what's this? And you've got no idea. And then there's some small tweak and it's zoomed out and "aha, of course". Now the interesting things about those is partly it's a perception thing that your eye can get fooled. But they're often deliberately chosen at a very unusual eye position. So you may take a photograph of something so that certain things line up, that then make it confusingly look like something else. And you only have to perhaps move your head a little bit.

You know, if it was a real thing that you were looking at, you would move it, you would move your head. And suddenly it would all make sense. Our perception systems are designed for the real world. They're not designed for static images on screens. We've learned to deal with them – I said culture comes in here. So our brain is trying to make sense of this all the time. And that can sometimes mean you can get these optical illusions that can be confusing. But also that can work to our advantage because we're trying to make sense.

So this is something you can think about in design, both things that can go wrong, but also things that can go right.

What I want to focus is four different reasons for why externalization can be powerful.  Some of you might remember – this was quite a few years ago now, so some of you might be too young to remember, but during the war in Afghanistan Rumsfeld talked about three kinds of knowns. He talked about the *known knowns*:

the things you know you know; the *known unknowns*: the things you know you don't know, and the *unknown unknowns*: the things you don't know – you don't even know that you don't know them. Now, Rumsfeld's answer to the last of these was just to blow everyone to pieces. We're not going to suggest this as a general  user experience and user interface design strategy. However, what I want to focus is – he mentioned three things here, and there's a missing one. So, we have the known knowns,

the known unknowns,  the unknown unknowns, what about *unknown knowns*? Now, I'm going to say these are the most interesting things. These are things you know – you or perhaps people you're working with know – *tacit things*, but they *aren't aware that you know*. So, this is true of your own understanding of a problem area. Or it might be true as you talk to your users or your stakeholders or your clients,

to actually uncover the things that they know and yet don't know that they know. And externalization is about that process that takes the unknown knowns *and turns them into known knowns*. There's all sorts of different kinds of externalization used in design, ways we put our knowledge into the world. There's – I mean, classic is *drawing and sketches*. You can have *models* and actually models in multiple senses

– models in the sense of physical models of buildings and things like that, but also mathematical models and models of that kind; *diagrams*, formal diagrams; *mathematical formulae* – again, I mentioned before my first love was mathematics, so I like my formulae; *spoken words* – so, things we say; criteria words, dimension words, abstract ideas – sort of adjectives often, even, or adverbs. People will use those words to talk about their area, but not

– if you ask them what are the critical features, what are the critical criteria, they wouldn't necessarily be able to name them. So, often the words we speak externalize things; so, listening to other people. Also learning to listen to yourself – often easier with *written words*, whether again it's your own or somebody else's because then you can analyze this. It's a classic thing to do, for traditionally in requirements engineering but also for working out what – to understand a new situation is to look at the written documentation, the manuals,

the procedures – perhaps things that have been written about an area. And people often do things like noun-verb analysis, of pulling out, trying to pull out critical things. Do that to your own writing about something in order to try and understand what you know about it. *Computer programs*, software embodies tacit knowledge in physical things or at least digital things that you can analyze. *Acting out* – talking about the sort of internet-enabled Swiss Army Knife,

and it was only as I acted that I made it external, did something physically, that I saw there was a problem with covering where you might decide to put your little screen. So, why are you doing this? I've sort of already given this a little bit away with the Rumsfeld one, but I'm going to give you four different reasons for why externalization can be powerful. So, we're going to talk about the *informational use*, which is about existing ideas; *formational use* – the way that externalization creates new ideas;

*transformational*, which is actually using the external thing to do stuff; and in the end *transcendental*, but not necessarily in the way that you might hear the word 'transcendental' used. So, first of all, *informational*. Now, the informational way of using ideas is, shall we say, the *classic* form of externalization. I've got books behind me, so I'm going to pull a book off the shelf. So, this is *Human-Computer Interaction* by Dix, Finlay, Abowd and Beale.

So, the informational model is probably  the way you'd normally think about writing. You know – we knew things about human-computer interaction. We wrote them into the book; people buy the book; people read the book, and then  into their heads... human-computer interaction. So, an idea in this person's head – the person on the left here is our

font of knowledge on the subject; writes it down in some way, and then the person on the right reads or sees that, or it may be a video lecture like this – it doesn't have to be necessarily written. And the information is passed on. You notice I've drawn them slightly  different in each state because typically when things pass on, the idea that gets written  down may be not quite the abstract thought in the head, and what the person understands when they read what's written down

may be different again. But the idea is that you're passing knowledge from mind to mind. There are the things I know and then I want to pass on to you. That's a classic model of information transfer model of stuff. So, that happens – you know – so, that is a thing that occurs. And in some sense that's what I'm doing here. I've thought about these issues; I've drawn slides. So, probably things are happening dynamically, but a lot of it I knew already;

I've transferred into slides, into ideas that I'm going to pass on and talking to them. So, that's the informational model, a classic model of what you do with external representations. However, if as a designer you've sketched things, if you've ever done creative writing and you've had a character, you know it's not all that. So... you sort of want the character to do something and you can't get the character to do something, because  you know the character wouldn't do it that way.

Suddenly, as things flow out, they become more  concrete, more clear, more explicit. So, as you draw your sketch of that user interface, you suddenly start – things become clearer that before were very sort of fuzzy in your head. You sort of half knew them, and the act of putting them out there makes them explicit. So, when you – even in something  like this video, I've drawn slides for it, and in drawing those slides I have made my thoughts much more explicit.

You've probably been in that situation where you write something down and then you suddenly think, 'I didn't know that before.' Perhaps you notice afterwards; you've  written it down; you read it back and you think, 'Did I write that? I didn't know that.' A classic thing – I said, talk to any writer, but you'll have almost certainly experienced it yourself during your design in sketching or whatever if you're more of a

graphic designer, if you're an interaction designer, the way you've done it. So, here what's happening is you've  got this fuzzy thought in your head; you externalize it in some way – you write it down, and during that process of externalization, you're forced to both think about the idea and make it more explicit, more concrete. So, actually what's happening is the idea in your head is being transformed from something a bit fuzzy to being something that you've actually got a bit more of a handle on.

So, that's *formational understanding* coming through externalization. So, let's go one step further – *transformational*. So, having externalized... and during that process almost certainly made some of your thoughts much more concrete and explicit, can you use those materials to do stuff? One that I've been using recently which is in this – that is using again the *physical representation*

as a way of *thinking* about stuff is with  producing slides for talks and for videos. So, what I've been doing is I'll print out  my initial ideas of slides. So, what I then do is – I normally do this with scissors and I forgot to bring my scissors with me into the... perhaps I'll tear it and hopefully tear  it without... and on the line; I'll tear it again. So, I end up with each slide on a piece of paper. And in true Blue Peter fashion...  

here are some I prepared earlier. And then what I find is that when I've done that, it's much easier – I certainly find it  easier to spread them all out on the table top and then sort them in and put  them into different categories. And then, once I'm sort of satisfied with them, I might – and I was trying to see if any of these

have got notes on – I might write little notes on them. I'm just thumbing through to see if I can find one with a note on – like there, I've scribbled some notes and comments into that one. Sometimes, I'll (inaudible) – I might cut out a few blank ones, and if there are gaps, I might write some notes for a new slide to go in there, put them into there, or once I've organized them, put them into little packets like this, and suddenly now I have my talk all organized. I know what I've got; I know where the  gaps are. I don't feel worried about slides

that I think I might use, because if they're sitting at the side of the table, I've not lost them, even if I decide not to put them in. Very powerful – well, I find it powerful: *reasoning using the physical representation*, chopping it up. Now, in principle, I could do this all in PowerPoint. I could pull slides around, sort them around. But there's something about having them there that certainly makes it feel easier to do and for a lot of purposes. So, you're thinking actually physically  using the materials

in order to do stuff. So, that's *transformational use of externalization*. Oh yes, that's right – I almost forgot; you might have heard – if not, I'll explain it to you then the ideas of *external distributed cognition*. Some people talk about *embodied thinking*  or *embodied cognition*: the idea that when we think, our thoughts are part of the world *outside*; they're not just in our heads. They're part of our interactions with the world.  Now, that's something important as a designer, to think about in relation to your users – you know – that they're not necessarily

a pure cognitive creature. They're a creature with hands and arms, and there's a world they're living in. And we often *offload* – people talk about 'offloading cognition'. Sometimes that's about *memory*: you know – the fact that you don't remember all your telephone numbers but they're in your telephone. Sometimes it's about offloading *thought processes*.  So, if I want to do a complex sum, I don't try and do it all in my head. It's complex. I write it down and scribble and draw,

sometimes draw little pictures and stuff like that. And so, effectively our cognition is not just in our head, but it's part of the way we relate to our environment. So, that's something to think about as a designer because your users are doing it, but also as *part of your design*. You are thinking externally embodied, and you can deliberately create techniques to help you do that.

So, the last kind of externalization we'll talk  about is *transcendental externalization*. So, the idea that the – well, actually, I say it's 'kind'; it's not 'kind', really – they're all different things that happen as we externalize. So, this is where our *internal* thoughts and ideas become the *object* of thought. By naming it, by being able to talk about it, suddenly now I can talk about that as an issue. If it's in graphic design, at the point at which you label it and you say, 'Oh, actually, the reason why

I find that's – I dislike that image, I don't find that a powerful one; the reason I like that one instead is because of *balance*.' Or... back to the HCI book. Now, when we were first shown the example cover for this book, it was identical, except that the hand was the other way around. So, I was going to get the right one here. So, the hand is doing... *that* is what you're seeing the hand do.

But actually it did it that way round and at a slight angle. And I remember looking at it and thinking... there was something wrong with it. It didn't *feel* right. But I realized one of the things I didn't like was the fact that because the hand did that, it was sort of pointing

bottom right towards top left. In Western tradition – I mean, I'm not sure how this looks to somebody whose writing goes the other way around, but certainly in left-to-right writing, things that go up like that are seen as  *dynamic* – going up, in fact. Things that do *that*.... And the arrow was sort of – I mean, it wasn't so much pointing down, but it was doing the wrong (inaud.). Having *articulated* that, was able to then take that  back to design.

But having been able to *articulate*, to *say*, to give a *name* to what was going on, it was about dynamism, it was about the left-to-right movement, then we were able to discuss it. Now, *I* wasn't able to produce a better cover, but the person – actually, or probably could have done this (inaud.); all the person did, I think, was do a mirror flip of the image. But it was the ability to articulate it. So, once that vague concept in your head that just didn't feel right

was *articulated*, then it was something you could communicate to others, you could think about yourself and  think about alternative ways of doing it. Now, those of you if you've... worked  in a sort of more ... a design area that isn't necessarily user experience design, you might have come across the writings of Schon, *The Reflective Practitioner*. So, Don Schon, I mean it's a classic in the design literature; he looked at a number of designers in different areas. There was sort of urban planning design,

architectural design. And in particular, some of this was about the relation with each other, but some of it was the relation with students, and the way in which really top-end designers were *reflective*; they didn't just say, 'Ah, that's a good idea,' but they said, 'This is good *because*'; 'There's a problem here *because*' and they were looking in at their own thinking and their own way they're working in order to

– partly, I said in a teaching concept, to be able to pass that on to a student, or possibly to be able to talk to others. But also, by being – and go for this word – 'Reflective Practitioners', looking in at their own practice, they were able also to be able to lift their own practice to a different level. They were able to think about what they were doing, about what was good about it, what was bad about it

to improve it, to be able to go to new situations and perhaps do a little bit more handle turning in the sense of saying, 'Have I thought about these issues?' But by being reflective, taking what had been a tacit, albeit successful practice and lifting it up to a higher plane.

Sensory memory is that memory you have of the word you just heard spoken or when you glance and close your eyes and you can just about remember some of what was in front of you. And arguably you might not even think of that as memory. So it's sort of like a memory, but not memory, but it really does actually make a difference to your life.

The most obvious way in which sensory memory happens that you might notice is in visual imaging. So if you stare at an image, say a Red Cross on a green background and you stare at it and you stare at it, you stare at it for a long time. And then you look to a white background, you'll often see an afterimage. On the white background you see the opposite colors often. It's particularly true if it's going to be a bright image.

If it's dull, you don't tend to notice it; it's got to be quite a bright image. And the reason for that is most of our senses have a very low-level dulling. So it's actually we've got multiple kinds of sensory memory here, so there's a dulling. If you have the same thing happening all the time, most of our senses don't notice it after a while. They notice change. That's true of touch. If you're touching something for a long while, you can stop feeling it. And it's true of sounds. If you've got the same sound going all the time,

it just fades into the background. And it's true of our visual senses. Sometimes it's called *habituation*. Your senses get used to things, and it's the *change* that you notice most. You can almost say we only ever see change; that's not entirely true – we can focus. But to a large extent it's change which drives the sensory processes. This is used, whether it's a computer movie but if you look at old sort of Victorian toys,

and also think of cine film, it's still the way a lot of high-quality film is taken even in a digital age. What you have is a series of still images that go after each other, one after another. So in fact, if you see, just like a dot move across the screen, you see... You think you're seeing a movement, but actually what you've got is a series of still points of that dot as it moves.

If you're looking at a person – if you're looking at me moving now – every movement I make, you're seeing probably 25 frames a second, perhaps 30 frames a second. So basically lots of stills and your brain fills in the gaps and makes these persistent. So that's happening at the sensory level and it's sort of a memory-ish sort of effect. But then as things get deeper into your brain, most of your senses have a buffer, a temporal buffer.

So particularly this is true of your sight to support the *iconic memory*, so that business of not just the after effects, which is really about the sensors in your eye, but the fact that I said, you can close your eyes and I've got a pretty good idea of what's in front. It doesn't last very long, and it varies from person to person. That's quite crucial. We don't all experience that in the same way. You have this thing called an *echoic memory*, which is this aural thing.

And you might have noticed this where somebody says something to you and you think you didn't quite grasp it and you start to say "Pardon?" or "What did you say?" And then you realize, and it's because you've replayed it in your head and then the second time around, you actually heard it. Just like you might play a replay if you're watching a video on the computer or listening to it on the computer. Now again, that varies. A personal thing, I actually have great difficulty in reproducing a sound I hear.

I obviously have a slight failing, and somewhere between my echoic memory connects into my speech production – I can hear the sounds really well; I can produce the sounds pretty well, but I struggle to connect them up. So different people, you know, I wouldn't call that a disability. Perhaps it is, but we're different. We're very different in the way these things get together. But most of us have a level of these things. And then you also can get the same thing with haptic memory...

There's a sensitizing we talked [about], where if you've touched the same thing for a long time, you don't notice. It's the change you tend to feel as your finger moves over a rough patch. But also you have a sense of what you just felt. So if you feel something, it's your brain sort of remembers it for a moment after your body stops actually physically sensing it. However, these are really, really, really short term. They're just constantly getting overwritten. So it's a bit like having a loop on an old tape recorder

but one that's so short it has to be [re]written. So it is very simple. That echoic memory, that ability to say, "What did you say?" that... Almost to re-hear it disappears very, very rapidly. Now you might remember what they said and be able to replay that in your head. But the actual sound memory disappears exceedingly rapidly.