混合现实中的音频Audio in mixed reality

音频是混合现实中设计和生产力的必不可少部分。Audio is an essential part of design and productivity in mixed reality. 声音可以:Sound can:

  • 提高用户在手势和语音交互中的置信度。Increase user confidence in gesture and voice interactions.
  • 指导用户接下来的步骤。Guide users to next steps.
  • 有效地将虚拟对象与现实世界组合在一起。Effectively combine virtual objects with the real world.

混合现实耳机的低延迟标题跟踪(包括 HoloLens)支持高质量的基于 HRTF 的 spatialization。The low-latency head tracking of mixed reality headsets, including HoloLens, supports high-quality HRTF-based spatialization. 你可以在应用程序中 spatialize 音频,以执行以下操作:You can spatialize audio in your application to:

  • 注意视觉对象。Call attention to visual elements.
  • 帮助用户保持对真实世界环境的认知。Help users maintain awareness of their real-world surroundings.

噪声更深入地连接到混合现实世界。Acoustics more deeply connect holograms to the mixed-reality world. 它提供有关环境和对象状态的提示。It provides cues about the environment and object state.

请参阅 使用音频的设计的详细示例See detailed examples of design that uses audio.


设备支持Device support

功能Feature HoloLens (第一代) HoloLens (first gen) HoloLens 2HoloLens 2 沉浸式头戴显示设备Immersive headsets
空间化Spatialization ✔️✔️ ✔️✔️ ✔️✔️
Spatialization 硬件加速Spatialization hardware acceleration ✔️✔️

在混合现实中使用声音Use of sounds in mixed reality

混合现实中使用声音 需要不同于触摸和键盘和鼠标应用程序的方法。Use of sounds in mixed reality requires a different approach than in touch and keyboard-and-mouse applications. 关键的设计决策包括哪些声音要 spatialize 以及哪些 sonify 交互。Key sound design decisions include which sounds to spatialize and which interactions to sonify. 这些决策会大大影响用户的置信度、生产力和学习曲线。These decisions strongly effect user confidence, productivity, and learning curve.

案例研究Case studies

HoloTour 几乎使用户能够在世界各地旅游和历史站点。HoloTour virtually takes users to tourist and historical sites around the world. 请参阅 HoloTour 案例研究的 声音设计See the Sound design for HoloTour case study. 使用特殊的麦克风和渲染设置来捕获使用者空间。A special microphone and rendering setup were used to capture the subject spaces.

RoboRaid 是一种射击的高能耗。RoboRaid is a high-energy shooter for HoloLens. RoboRaid 案例研究的 声音设计 介绍了用于确保空间音频最大效果的设计选择。The Sound design for RoboRaid case study describes the design choices that were made to ensure spatial audio was used to the fullest dramatic effect.

空间化Spatialization

Spatialization 是空间音频的方向性组件。Spatialization is the directional component of spatial audio. 对于7.1 家庭影院设置,spatialization 与 loudspeakers 之间的平移非常简单。For a 7.1 home theater setup, spatialization is as simple as panning between loudspeakers. 但对于混合现实中的耳机,使用基于 HRTF 的技术是非常重要的。But for headphones in mixed reality, it's essential to use an HRTF-based technology for accuracy and comfort. Windows 提供基于 HRTF 的 spatialization,这种支持在 HoloLens 2 上是硬件加速的。Windows offers HRTF-based spatialization, and this support is hardware-accelerated on HoloLens 2.


我应该 spatialize 吗?Should I spatialize?

Spatialization 可以改进混合现实应用程序中的许多声音。Spatialization can improve many sounds in mixed-reality applications. Spatialization 从侦听器的头中取出声音,并将其放在世界各地。Spatialization takes a sound out of the listener's head and places it in the world. 有关在应用程序中有效使用 spatialization 的建议,请参阅 空间音效设计For suggestions on effective use of spatialization in your application, see Spatial sound design.

Spatializer 个性化Spatializer personalization

HRTFs 跨频率范围控制耳之间的级别和阶段差异。HRTFs manipulate the level and phase differences between ears across the frequency spectrum. 它们基于物理模型和 torso 和 ear) (pinnae 的度量。They're based on physical models and measurements of human head, torso, and ear shapes (pinnae). 我们大脑对这些差异做出了响应,以提供合理的声音方向。Our brains respond to these differences to provide perceived direction in sound.

每个人都有独特的耳形状、头大小和耳位置。Every individual has a unique ear shape, head size, and ear position. 因此,最佳 HRTFs 与你相符。So the best HRTFs conform to you. 为了提高 spatialization 准确度,HoloLens 使用 pupilary 距离 (IPD) ,以调整头大小的 HRTFs。To increase spatialization accuracy, HoloLens uses your inter-pupilary distance (IPD) from the headset displays to adjust the HRTFs for your head size.

Spatializer 平台支持Spatializer platform support

Windows 通过 ISPATIALAUDIOCLIENT API提供 spatialization,包括 HRTFs。Windows offers spatialization, including HRTFs, via the ISpatialAudioClient API. 此 API 向应用程序公开 HoloLens 2 HRTF 硬件加速。This API exposes the HoloLens 2 HRTF hardware acceleration to applications.

Spatializer 中间件支持Spatializer middleware support

以下第三方音频引擎提供对 Windows "HRTFs 的支持。Support for Windows' HRTFs is available for the following third-party audio engines.

声音Acoustics

空间音频约为方向。Spatial audio is about more than direction. 其他维度包括封闭、障碍、回音、portaling 和源建模。Other dimensions include occlusion, obstruction, reverb, portaling, and source modeling. 这些维度统称为 " 噪声"。Collectively these dimensions are referred to as acoustics. 如果没有噪声,spatialized 声音就会缺少距离。Without acoustics, spatialized sounds lack perceived distance.

噪声治疗范围从简单到复杂。Acoustics treatments range from simple to complex. 您可以使用任何音频引擎支持的回音,将 spatialized 的声音推送到侦听器的环境中。You can use a reverb that's supported by any audio engine to push spatialized sounds into the environment of the listener. 诸如 项目噪声 等噪声系统提供更丰富且更具吸引力的噪声处理。Acoustics systems such as Project Acoustics provide richer and more compelling acoustics treatment. 项目噪声可以为声音上的墙壁、门和其他场景几何的效果建模。Project Acoustics can model the effect of walls, doors, and other scene geometry on a sound. 这是一个有效的选项,适用于在开发时了解相关场景几何图形的情况。It's an effective option for cases where the relevant scene geometry is known at development time.

后续步骤Next steps