Literature on Mixed Reality (MR)

Here’s the entire UX literature on Mixed Reality (MR) by the Interaction Design Foundation, collated in one place:

Beyond AR vs. VR: What is the Difference between AR vs. MR vs. VR vs. XR?

There is a fair amount of confusion about the differences between the terms virtual reality (VR), augmented reality (AR), mixed reality (MR), and extended reality (XR). While most people have stuck with AR and VR to describe the different technologies, these terms are not enough to fully comprehend the extent of these technologies and harness their potential. That’s why we’re going to explore them in detail now.

According to research by MarketsandMarkets, the market size of all extended reality technologies combined is expected to reach USD 125.2 billion by 2026, at a compound annual growth rate (CAGR) of 30.6% between 2021 and 2026. As a UX designer, you have the chance to contribute to this technology revolution and help define how to take user experience to the next level.

The first step is to define each technology clearly. Once you understand the possibilities each one offers, you’ll be able to adapt your UX design knowledge to these new environments.

Let’s start by looking at the word they all have in common: “reality.” What does “reality” mean in the context of AR, MR, VR and XR?

What is Reality?

Have you ever been in an argument with someone over a specific color? Did it seem dark blue to you and black to the other person? To some extent, the way we process information and construct reality is unique to each one of us—our genetics, previous experiences, etc.—and shapes how we perceive the world.

Humans tend to confuse reality with the physical world and struggle to understand why virtual reality feels so real even when they know it is not.

“We have had people literally run out of the VR room, even though they know that what they are witnessing is not real”
— Mel Slater, Distinguished Scientist and VR pioneer, Institute of Neurosciences of the University of Barcelona

To understand these technologies, you need to understand reality as a construct that each of us makes based on what we perceive from our senses, whether what we perceive comes from the digital or the physical world. The same thing happens when we watch movies. Do you remember the last time you cried or laughed while watching a movie? Were you aware that what you were watching was not real? Yes. Did it make you cry or laugh anyway? If the movie was good, absolutely! Even though you knew you were watching fiction, it felt real at the time. Extended reality technologies add another layer to this phenomenon, which makes the experiences feel even more intense.

“It’s really important to understand we’re not seeing reality. We’re seeing a story that’s being created for us.”
— Patrick Cavanagh, Research Professor, Dartmouth College

When you wear a VR headset, you can feel present in a fully digital environment. The digital information that you perceive through your senses overpowers your reasoning “this is not real.” For your body, at that moment, it is real.

And for all these technologies to work, they all need to feel real. The difference between them is how much they rely on physical or digital elements.

What is the Virtuality Continuum?

The virtuality continuum—continuum being the critical word—contains the full spectrum of possibilities between the entirely physical world or real environment and the fully digital world or virtual environment. In a continuum, adjacent parts are almost indistinguishable, but the extremes are very different. The researchers Paul Milgram and Fumio Kishino first introduced the virtuality continuum or reality-virtuality continuum concept in 1994.

Representation of the virtuality continuum (adapted from Milgram; Kishino, 1994).

The virtuality continuum is a theoretical framework that can help you visualize and understand the differences between the various technologies that exist today and those that are yet to be invented. For example, you can simplify the concept and think of a spectrum of immersion, where one end is low immersion and the other end is high immersion. Then, you can situate the technologies according to the degree of immersion they provide.

Representation of current XR technologies according to the spectrum of immersion.

Sometimes the exact limits of the various technologies are not a hundred percent clear, and they may overlap. However, the critical part is that you understand the region they occupy in the spectrum. This exercise will help you clarify the differences between all these technologies.

What is Extended Reality (XR)?

Extended reality (XR) is an umbrella term that encompasses any sort of technology that alters reality by adding digital elements to the physical or real-world environment by any extent, blurring the line between the physical and the digital world.

XR includes AR, MR, VR, and any technology—even those that have yet to be developed—situated at any point of the virtuality continuum.

The term XR includes AR, MR, VR, and any technology that blends the physical and the digital world.

Bear in mind that XR technologies keep evolving, and their full potential remains to be seen. There are still many things to discover about how users can better interact with them and achieve the best results. For a UX designer specializing in XR experiences, the difficulty lies in the lack of defined standards. An excellent place to start is to keep experiences simple and push the envelope a little bit at a time. In the case of new technologies, usability testing will be your best friend.

What is Augmented Reality (AR)?

Augmented reality (AR) is a technology that allows the superposition of digital elements into the real-world environment. In the AR experience, you can see a composite view of physical or real-world elements and digital elements. While some AR experiences may offer a certain degree of interaction between physical and virtual elements, typically, there is limited to no direct interaction between the digital and physical world components.

AR experiences are close to the physical world end of the virtuality continuum. The ability to overlay digital objects onto the physical world is revolutionizing many industries such as gaming, education, healthcare, and manufacturing. For example, have you ever been to the doctor for a blood test, and the nurse couldn’t find your vein? It can be excruciating. What if AR technology could help with that?

AccuVein uses projection-based AR technology and a laser-based scanner to convert the heat signature of a patient’s veins into an image superimposed on the skin, making the veins easier for clinicians to locate. This AR technology increases the likelihood of a successful first-time injection by 350%.

© AccuVein, Fair-Use (link)

As a UX designer specializing in AR experiences, you have the chance to go beyond the rectangular square of a screen and contribute to a revolution that will continue to change the way we interact with digital products. However, you’ll need to be aware of additional parameters to deliver a good UX that you may not be used to, like physical constraints and safety issues.

Get your free template for “AR Usability Plan Example”

Secure form

Name Please provide your name.

We respect your privacy

Please provide a valid email address.

Get free UX design learning material every week

316,660 designers enjoy our newsletter—sure you don’t want to receive it?

What is Mixed Reality (MR)?

Mixed reality (MR) is a technology that allows not only the superposition of digital elements into the real-world environment but also their interaction. In the MR experience, the user can see and interact with both the digital elements and the physical ones. Therefore, MR experiences get input from the environment and will change according to it.

In MR experiences the user can interact with both digital and physical elements. MR differs from AR—where digital and physical elements don’t interact— and VR—where the physical or real world is completely blocked out.

MR experiences cover the center of the virtuality continuum.

The concept of mixed reality can be a bit difficult to grasp; watch the video below to see an example of an MR experience.

This video shows how MR experiences blend the physical and digital worlds.

As a UX designer specializing in MR, you’ll have to master all the possibilities that MR technologies have to offer. Immersive experiences add a new layer to the user experience and require you to learn continuously and stay up to date to deliver an excellent user experience.

What is Virtual Reality (VR)?

Virtual reality (VR) is a technology that allows the creation of a fully-immersive digital environment. In VR experiences, the physical or real-world environment is entirely blocked out.

VR experiences are located at the fully virtual extreme of the virtuality continuum.

Many people struggle with the fact that VR experiences generate true emotional responses even if we know it’s “fake.” Keep in mind that humans construct reality from the information they receive from their senses, and this is why, even if we are aware that we have a fully digital experience, our bodies respond in the same way. Typically, VR takes advantage of the visual and auditory systems. However, there is an even greater sense of presence and immersion if you add other senses. Have you heard about the “Walk the Plank” challenge? It is a VR experience where you enter an elevator and get out at the top of a skyscraper. Then you’re asked to “walk a plank.” Just with auditory and visual stimulation, many people (especially those who suffer from a fear of heights) can’t do it, even though they are aware that what they see and hear is not real. However, if you add a physical plank to the setup and thus activate the sense of touch, even fewer people will be able to do it.

In this video, you can see how football players try the “Walk the Plank” VR experience with a physical plank and struggle to complete the experience, even though they are aware that the heights are not “real.”

The more coherent information we get through our senses, the more immersive the experience. As a UX designer, you’ll have to consider the different dimensions of the experience and the particularities of VR technology to create the best possible experience for your user.

Get your free template for “Storyboard for VR”

Secure form

Name Please provide your name.

We respect your privacy

Please provide a valid email address.

Get free UX design learning material every week

316,660 designers enjoy our newsletter—sure you don’t want to receive it?

The Take Away

There are many technologies that alter our perception of reality by adding digital elements to the physical world to a greater or lesser extent. All these technologies can be situated at some point of the virtuality continuum according to the extent that the digital elements block the real-world environment and their interactive capabilities. If you have the image of the virtuality continuum in your head, you’ll be able to clearly see the difference between each type of technology.

The most important terms to learn to harness the potential of these new technologies are:

Augmented reality (AR): a view of the real world—physical world—with an overlay of digital elements.
Mixed Reality (MR): a view of the real world—physical world—with an overlay of digital elements where physical and digital elements can interact.
Virtual reality (VR): a fully-immersive digital environment.
Extended reality (XR): an umbrella term that covers all these different technologies, including AR, MR, and VR.

As all these technologies keep evolving and new ones emerge, UX designers need to keep learning and understanding the characteristics of each one to deliver the best possible user experience. If you can clearly distinguish AR, MR and VR and any other XR technology, you’ll be able to design experiences that take advantage of each technology and give an outstanding boost to your UX career!

References and Where to Learn More

Learn how to design your own XR experiences with our course: How to Design for Augmented and Virtual Reality.

Watch the How To Influence Behavior Through Virtual Reality Narratives on-demand Master Class by VR pioneer Mel Slater.

Vi, S. (2022). Exploring User Experience Guidelines for Designing HMD Extended Reality Applications.

The way you see colour depends on what language you speak. (2022). Aina Casaponsa, Panos Athanasopoulos.

UX for XR by Cornel Hillmann.

Learn more about Mixed Reality (MR)

Take a deep dive into Mixed Reality (MR) with our course UX Design for Virtual Reality .

Virtual reality is a multidimensional universe that invites you to bring stories to life, transform digital interactions, educate with impact and create user-centric and unforgettable experiences. This course equips you with the skills and knowledge to embrace the possibilities and navigate the challenges of virtual reality.

UX Design for Virtual Reality is taught by UX expert Frank Spillers, CEO and founder of the renowned UX consultancy Experience Dynamics. Frank is an expert in the field of VR and AR, and has 22 years of UX experience with Fortune 500 clients including Nike, Intel, Microsoft, HP, and Capital One.

In UX Design for Virtual Reality, you’ll learn how to create your own successful VR experience through UX design. Informed by technological developments, UX design principles and VR best practices, explore the entire VR design process, from concept to implementation. Apply your newfound skills and knowledge immediately though practical and enjoyable exercises.

In lesson 1, you’ll immerse yourself in the origins and future potential of VR and you’ll learn how the core principles of UX design apply to VR.

In lesson 2, you’ll learn about user research methods, custom-tailored for the intricacies of VR.

In lesson 3, you’ll investigate immersion and presence and explore narrative, motion and sounds as design tools.

In lesson 4, you’ll delve into interface and interaction design to create your own user-friendly, compelling and comfortable VR experiences.

In lesson 5, you’ll gain insights into prototyping, testing, implementing VR experiences, and conducting thorough evaluations.

After each lesson you’ll have the chance to put what you’ve learned into practice with a practical portfolio exercise. Once you’ve completed the course, you’ll have a case study to add to your UX portfolio. This case study will be pivotal in your transition from 2D designer to 3D designer.

UX Design for Virtual Reality

Closes in

-10

days

-15

hrs

-53

mins

-23

secs

booked

All open-source articles on Mixed Reality (MR)