混合現實的音效Audio in mixed reality

音訊是設計和生產力在混合現實中不可或缺的一部分。Audio is an essential part of design and productivity in mixed reality. 音效可以:Sound can:

  • 提高筆勢和語音互動的使用者信賴度。Increase user confidence in gesture and voice interactions.
  • 引導使用者進行後續步驟。Guide users to next steps.
  • 有效地結合虛擬物件與真實世界。Effectively combine virtual objects with the real world.

混合現實耳機的低延遲 head 追蹤(包括 HoloLens)支援高品質的 HRTF 型 spatialization。The low-latency head tracking of mixed reality headsets, including HoloLens, supports high-quality HRTF-based spatialization. 您可以在應用程式中 spatialize 音訊,以:You can spatialize audio in your application to:

  • 注意視覺元素。Call attention to visual elements.
  • 協助使用者維護其真實世界周圍的認知。Help users maintain awareness of their real-world surroundings.

聲場更深入地連接全像混合現實世界的影像。Acoustics more deeply connect holograms to the mixed-reality world. 它會提供有關環境和物件狀態的提示。It provides cues about the environment and object state.

請參閱 使用音訊之設計的詳細範例。See detailed examples of design that uses audio.


裝置支援Device support

功能Feature HoloLens (第一代) HoloLens (first gen) HoloLens 2HoloLens 2 沉浸式頭戴裝置Immersive headsets
空間化Spatialization ✔️✔️ ✔️✔️ ✔️✔️
Spatialization 硬體加速Spatialization hardware acceleration ✔️✔️

在混合現實中使用音效Use of sounds in mixed reality

在混合現實中使用 音效需要的方法與觸控和鍵盤和滑鼠應用程式不同。Use of sounds in mixed reality requires a different approach than in touch and keyboard-and-mouse applications. 主要的音效設計決策包括要 spatialize 的音效,以及要 sonify 的互動。Key sound design decisions include which sounds to spatialize and which interactions to sonify. 這些決策會大幅影響使用者信賴度、生產力和學習曲線。These decisions strongly effect user confidence, productivity, and learning curve.

案例研究Case studies

HoloTour 幾乎會讓使用者在世界各地旅遊和歷程記錄網站。HoloTour virtually takes users to tourist and historical sites around the world. 請參閱 HoloTour 案例研究的 音效設計See the Sound design for HoloTour case study. 用來捕捉主體空間的特殊麥克風和轉譯設定。A special microphone and rendering setup were used to capture the subject spaces.

RoboRaid 是適用于 HoloLens 的高能源射擊。RoboRaid is a high-energy shooter for HoloLens. RoboRaid 案例研究的 音效設計 說明了可確保使用空間音訊來發揮最大效果的設計選擇。The Sound design for RoboRaid case study describes the design choices that were made to ensure spatial audio was used to the fullest dramatic effect.

空間化Spatialization

Spatialization 是空間音訊的方向元件。Spatialization is the directional component of spatial audio. 若為7.1 家用劇院設定,spatialization 就像在 loudspeakers 之間移動一樣簡單。For a 7.1 home theater setup, spatialization is as simple as panning between loudspeakers. 但對於混合式的耳機,使用 HRTF 型技術很重要且舒適。But for headphones in mixed reality, it's essential to use an HRTF-based technology for accuracy and comfort. Windows 提供以 HRTF 為基礎的 spatialization,這項支援在 HoloLens 2 的硬體加速。Windows offers HRTF-based spatialization, and this support is hardware-accelerated on HoloLens 2.


我應該 spatialize 嗎?Should I spatialize?

Spatialization 可以改善混合現實應用程式中的許多音效。Spatialization can improve many sounds in mixed-reality applications. Spatialization 會從接聽程式的頭部取出聲音,將它放在世界各地。Spatialization takes a sound out of the listener's head and places it in the world. 如需在應用程式中有效使用 spatialization 的建議,請參閱 空間音效設計For suggestions on effective use of spatialization in your application, see Spatial sound design.

空間定位器個人化Spatializer personalization

Hrtf 會操控各頻間的耳之間的層級和階段差異。HRTFs manipulate the level and phase differences between ears across the frequency spectrum. 它們是以人類 head、torso 和 ear (pinnae) 的實體模型和測量為基礎。They're based on physical models and measurements of human head, torso, and ear shapes (pinnae). 我們的大腦會回應這些差異,以提供音效的觀察方向。Our brains respond to these differences to provide perceived direction in sound.

每個人都有獨特的 ear 形狀、前端大小和 ear 定位。Every individual has a unique ear shape, head size, and ear position. 因此最適合您的 Hrtf。So the best HRTFs conform to you. 為了提高 spatialization 準確度,HoloLens 會使用您的 pupilary 距離,從耳機顯示器 (IPD) ,以調整您的前端大小 Hrtf。To increase spatialization accuracy, HoloLens uses your inter-pupilary distance (IPD) from the headset displays to adjust the HRTFs for your head size.

空間定位器平臺支援Spatializer platform support

Windows 透過 ISPATIALAUDIOCLIENT API提供 spatialization,包括 hrtf。Windows offers spatialization, including HRTFs, via the ISpatialAudioClient API. 此 API 會向應用程式公開 HoloLens 2 HRTF 硬體加速。This API exposes the HoloLens 2 HRTF hardware acceleration to applications.

空間定位器中介軟體支援Spatializer middleware support

下列協力廠商音訊引擎可使用 Windows 的 Hrtf 支援。Support for Windows' HRTFs is available for the following third-party audio engines.

聲學Acoustics

空間音訊大約是方向。Spatial audio is about more than direction. 其他維度包括遮蔽、障礙物、回音、傳送和來源模型。Other dimensions include occlusion, obstruction, reverb, portaling, and source modeling. 這些維度 統稱為聲場。Collectively these dimensions are referred to as acoustics. 在沒有聲場的情況下,hrtf 音效缺乏認知距離。Without acoustics, spatialized sounds lack perceived distance.

聲場的治療範圍從簡單到複雜。Acoustics treatments range from simple to complex. 您可以使用任何音訊引擎所支援的回音,將 hrtf 音效推送至接聽程式的環境。You can use a reverb that's supported by any audio engine to push spatialized sounds into the environment of the listener. 聲場系統(例如 聲場專案 )提供更豐富且更吸引人的聲場處理。Acoustics systems such as Project Acoustics provide richer and more compelling acoustics treatment. 聲場專案可以建立音效上牆、大門和其他場景幾何的效果模型。Project Acoustics can model the effect of walls, doors, and other scene geometry on a sound. 在開發階段已知相關場景幾何的情況下,這是有效的選項。It's an effective option for cases where the relevant scene geometry is known at development time.

後續步驟Next steps