Spatial Audio

Spatial audio, also known as 3D audio, refers to sound design and technology that mimics the natural way humans perceive sound in a three-dimensional space. It creates the illusion of sounds coming from different directions and distances, which provides a more immersive and realistic auditory experience. Spatial audio is often used in virtual reality (VR), gaming, and cinema to enhance the sense of presence and immersion in a digital environment.

Spatial Audio in Virtual Reality

Spatial audio in VR and other mediums differs from traditional stereo sound as it replicates how we hear sounds in the real world. Traditional audio offers a left-right dynamic.

In this video, UX Design Consultant Frank Spillers discusses the importance of spatial audio in VR.

How Does Spatial Audio Work

Spatial audio in VR adds to the immersion of an experience by simulating how people hear sounds in real life. Here's a breakdown of how it functions:

1. 3D Sound Simulation: Spatial audio replicates how sound behaves in a three-dimensional space. This means sounds in VR can seem to come from above, below, behind, in front of, or beside the listener. It's different from traditional stereo audio, which can only convey left and right directions.

2. Head-Related Transfer Function (HRTF): One of the key technologies behind spatial audio is the HRTF. It's a set of algorithms that mimic how our ears receive sounds from different directions and distances. HRTFs account for the fact that sounds are altered by the shape of our ears, head, and torso before reaching our eardrums. This alteration helps our brain understand the sound's location.

3. Binaural Recording Techniques: To create spatial audio content, binaural recording is often used. This involves placing two microphones in the ears of a dummy head or a human to record sound exactly as it would be heard by a person. When played back through headphones, this creates a highly realistic and spatially accurate sound experience.

   This video explores what binaural sound is and how it’s recorded.

   
   4. Directional Audio Cues: In VR, as the user moves their head or navigates through the virtual environment, the spatial audio adjusts in real-time. If a sound source is to the right of the user and they turn their head towards it, the sound will naturally seem to come from straight ahead, just as it would in real life.

4. Distance and Environment Effects: Spatial audio also simulates how sound changes with distance and in different environments. Sounds can become softer, more muffled, or change in quality as they get further away or as the environment changes (like echoing in a large hall or being absorbed in a densely furnished room).

5. Integration with VR Systems: Spatial audio is tightly integrated with software and hardware (like a VR headset). The VR system tracks the user's head movements and adjusts the audio in real-time to maintain the directionality and realism of sounds, enhancing the overall sense of presence in the virtual world.

What Are Diegetic and Non-Diegetic Sounds 

Diegetic sound and non-diegetic are different types of audio elements in virtual or digital worlds, like film and VR experiences. 

Diegetic Sound

Diegetic sound refers to audio elements that are part of the narrative world of the film, experience, or game. This means that the sound sources are visible on the “screen” or implied to be present by the action of the narrative. 

Diegetic sounds help to create a more immersive and believable world for the audience or user, as they mimic the audio experience of real life. 

Examples include:

1. Dialogue: Spoken words by the characters.

2. On-screen Actions: Sounds made by objects or actions visible on screen, like footsteps, doors closing, or cars passing.

3. Background Noise: Sounds originating from the setting or environment, such as birds chirping in a forest scene, or the hum of a spaceship in a sci-fi movie.

Non-Diegetic Sound

Non-diegetic sound are audio elements that are not part of the narrative world and are added for the user or audience's benefit. They’re often used to enhance the emotional or narrative impact and can be powerful tools in storytelling.

Examples include:

1. Musical Score: Background music that sets the mood or tone but is not being played within the story world (like a character turning on a radio).

2. Narration: Voiceovers that provide additional context or commentary but are not part of the dialogue within the story.

3. Sound Effects: Artificially added sounds to enhance the storytelling, such as dramatic stings in a thriller to heighten suspense.

How Spatial Audio Affects User Experience

Spatial audio has a significant impact on user experience, particularly in immersive environments like VR. Here's how it enhances the user experience:

1. Increased Immersion: Spatial audio replicates how we hear sounds in the real world. This realism helps to create a more immersive environment which makes digital experiences feel more lifelike. In VR, for instance, the user feels more present in the virtual world when the audio cues align with what they see.

2. Enhanced Spatial Awareness: With spatial audio, users can perceive the location of sound sources within a virtual environment. This directional audio cue is crucial in applications like VR gaming or training simulations as users need to understand the virtual environment.

3. Improved Navigation and Interaction: In interactive environments, spatial audio can guide users toward specific locations or alert them to actions happening off-screen or out of sight which adds to their sense of agency. This can be particularly useful in complex or dynamic VR and augmented reality (AR) settings.

4. Emotional and Narrative Depth: Just like in cinema, spatial audio in VR and AR can be used to evoke emotions and enhance storytelling. The direction and nature of sounds can build tension, convey mood, or signal changes in the narrative.

   Half-Life: Alyx uses spatial audio to elevate the overall game experience and deepen the impact of the narrative. Watch some of the gameplay in this next video: 

5. Increased Accessibility: For users with visual impairments, spatial audio can provide important contextual information about their environment, helping them to navigate and interact more effectively.

6. Better Communication in Social VR: In social VR applications, spatial audio allows for more natural conversations. Users can perceive who is speaking to them and from where, similar to a real-world group setting.

7. Enhanced Gameplay: In gaming, spatial audio contributes to both the strategy and enjoyment of the game. Players can locate other characters or actions by sound, providing a tactical advantage and a more engaging experience.

8. Reduced Cognitive Load: Accurate audio cues in a virtual environment can make it easier for users to process information, as they can use sound to understand and interact with the environment without relying solely on visual cues.

How to Implement Spatial Audio

Spatial audio or sound design for VR requires technical understanding and creative application. Here's how designers can effectively integrate spatial audio into their VR projects:

Understand the Technology: Designers should familiarize themselves with spatial audio technology. Tools and platforms like Oculus Spatializer, Steam Audio, or Google Resonance Audio offer specialized features to create spatial audio in VR.

Use the Right Equipment for Recording: Designers should consider using binaural microphones to record spatial audio. This setup captures the nuances of how sound interacts with the environment and the human body.

Incorporate Environmental Acoustics: The audio experience should reflect the virtual environment's characteristics. For instance, sounds should echo in a cavern or be muffled if coming from another room.

Integrate Audio with Visuals: The spatial audio cues should align with the visual elements. For example, if a character is speaking to the user from the right side, the sound should come from that direction.

Dynamic Audio Adjustment: Dynamic adjustments to the audio should be implemented based on the user's interactions and movements within the VR environment. As the user moves or turns their head, the audio sources should respond accordingly in real-time.

Effective Spatial Audio Examples

Spatial audio is already being used in powerful and significant ways, here are some real-world examples of spatial audio in practice: 

1. VR Games: Games like Half-Life: Alyx and Boneworks use spatial audio effectively to enhance immersion. In these games, players can discern the direction of threats and actions based on sound, which adds depth to gameplay.

2. Educational VR Experiences: Titans of Space, an educational tour of the solar system, uses spatial audio to make the experience more engaging. Users can hear narrations and sounds coming from specific points in the virtual space—this adds a layer of realism to the learning experience.

3. VR Concerts and Social Platforms: Platforms like Wave have hosted virtual concerts where spatial audio plays a key role. Artists perform in a virtual space, and the audio reflects the virtual environment, changing as users move around.

4. Cinematic VR: Wolves in the Walls, a VR narrative experience based on Neil Gaiman's work, uses spatial audio to enhance storytelling. The audio design helps to create a sense of presence and emotional connection with the virtual characters.

Learn More About Spatial Audio

Take our course on UX Design for Virtual Reality.

XR Today’s blog post, What is Spatial Audio? An Immersive Experience defines spatial audio and how it works in virtual environments.

Watch this video by Meta Quest about 3D audio.

Read this LinkedIn article, How do you design spatial audio for VR/AR games that are cross-platform and adaptive?