如何在混合现实应用程序中使用声音How to use sound in mixed-reality applications

您可以使用声音来通知和强化用户应用程序状态的心理模型。You can use sound to inform and reinforce the user's mental model of application state. 如果合适,请使用 spatialization,将声音置于混合现实世界中。Use spatialization, when appropriate, to place sounds in the mixed-reality world. 以这种方式连接听觉和视觉对象时,可以加深交互的直观特性并提高用户信心。When you connect the auditory and the visual in this way, you deepen the intuitive nature of interactions and increase user confidence.

何时添加声音When to add sounds

混合现实应用程序通常比2D 应用程序需要更高的声音,因为它们缺乏 tactile 接口。Mixed-reality applications often have a greater need for sound than 2D apps, because of their lack of a tactile interface. 在通知用户或强化交互时添加声音。Add sounds when they inform the user or reinforce interactions.

通知和强化Inform and reinforce

  • 对于不是由用户启动的事件(例如通知),请使用声音通知用户发生了更改。For events that aren't initiated by the user, such as notifications, use sound to inform the user that a change occurred.
  • 交互可能有多个阶段。Interactions may have several stages. 使用声音强化过渡过渡。Use sound to reinforce stage transitions.

请参阅下面的交互、事件和建议的声音特征的示例。See the following examples of interactions, events, and suggested sound characteristics.

运动挡板Exercise restraint

用户无限制音频信息。Users don't have an unlimited capacity for audio information.

  • 每个声音应传达特定的有价值信息。Each sound should communicate specific, valuable information.
  • 当应用播放声音以通知用户时,请暂时减少其他声音的音量。When your app plays a sound to inform the user, temporarily reduce the volume of other sounds.
  • 对于按钮悬停声音 (查看以下信息) ,添加时间延迟以防止过多的声音触发。For button hover sounds (see the following information), add a time delay to prevent excessive sound triggering.

不要完全依赖于声音Don't rely solely on sounds

使用良好的声音对于用户非常有用。Sounds that are used well are valuable to your users. 但请确保即使关闭声音,应用程序也能使用。But make sure your application is usable even with the sound turned off.

  • 用户可能会收到听力障碍。Users may be hearing impaired.
  • 您的应用程序可能会在更高的环境中使用。Your application may be used in a loud environment.
  • 用户可能会出于隐私目的或其他原因而禁用设备音频。Users may have privacy concerns or other reasons to disable device audio.

如何 sonify 交互How to sonify interactions

混合现实中的交互类型包括手势、直接操作和语音。Interaction types in mixed reality include gesture, direct manipulation, and voice. 使用以下建议特征为这些交互选择或设计声音。Use the following suggested characteristics to select or design sounds for these interactions.

手势交互Gesture interactions

在混合现实中,用户可以通过使用鼠标与按钮交互。In mixed reality, users may interact with buttons by using a mouse. 通常,当用户释放而不是按下按钮时,按钮操作会让用户有机会取消交互。Button actions generally occur when the user releases rather than presses the button to give the user a chance to cancel the interaction. 使用声音来强化这些阶段。Use sounds to reinforce these stages. 若要协助用户以较远的按钮为目标,还应考虑使用指针悬停声音。To assist users in targeting distant buttons, also consider using a pointer-hover sound.

  • 按钮-按下声音应为简短的 tactile "单击"。Button-press sounds should be a short, tactile "click."
    示例: MRTK_ButtonPress .wavExample: MRTK_ButtonPress.wav
  • 按钮-"unpress" 声音应具有类似的 tactile 感觉。Button-"unpress" sounds should have a similar tactile feel. 比按下声音更高的跨度强调完成的意义。A higher pitch than the press sound reinforces the sense of completion.
    示例: MRTK_ButtonUnpress .wavExample: MRTK_ButtonUnpress.wav
  • 对于 "悬停声音",请考虑使用细小的和非威胁的声音,如低频率 thud 或凹凸。For hover sounds, consider using a subtle and non-threatening sound, such as a low-frequency thud or bump.

直接操作Direct manipulation

在 HoloLens 2 上,已表述的手动跟踪支持直接操作用户界面元素。On HoloLens 2, articulated hand tracking supports direct manipulation of user-interface elements. 当没有其他物理反馈时,声音非常重要。Sounds are important when there's no other physical feedback.

按钮按下 声音非常重要,因为当用户到达关键笔画的底部时,用户不会收到任何其他指示。A button press sound is important because the user doesn't get any other indication when they reach the bottom of the key stroke. 关键旅行的声音指示器可以是小型、微妙和封闭像素。Sound indicators of key travel can be small, subtle, and occluded. 与手势交互一样,按钮按下会显示短暂的 tactile 声音,如单击。As with gesture interactions, button presses should get a short, tactile sound like a click. Unpresses 应该具有类似的单击声音,但具有凸起的音调。Unpresses should have a similar click sound but with raised pitch.

难以直观地确认获取或释放操作。It's difficult to visually confirm a grab or release action. 用户的手常常会受到任何视觉效果的影响,而 expression-bodied 对象却缺乏现实世界的 "抓取" 视觉模拟。The user's hand will often be in the way of any visual effect, and hard-bodied objects lack a real-world visual analog of "grabbing." 声音可以有效地传达成功的获取和释放交互。Sounds can effectively communicate successful grab and release interactions.

  • 抓取操作应该具有一个 muffled 的简短 tactile 声音,该声音将提示围绕对象的抓手指。Grab actions should have a short, somewhat-muffled tactile sound that prompts the idea of fingers closing around an object. 有时还会有一个 "whoosh" 声音,该声音会导致抓住声音传达手的运动。Sometimes there's also a "whoosh" sound that leads up to the grabbing sound to communicate the motion of the hand.
    示例: MRTK_Move_Start .wavExample: MRTK_Move_Start.wav
  • Release 操作应该获得类似的简短和 tactile 的声音。Release actions should get a similarly short and tactile sound. 它通常比抓住声音和反向顺序下的音调更小,但影响,然后使用 "whoosh" 传达对象正在进行的结算。It's usually lower pitched than the grab sound and in reverse order, with an impact and then a "whoosh" to communicate that the object is settling into place.
    示例: MRTK_Move_End .wavExample: MRTK_Move_End.wav

绘图 交互应会获得持久的循环声音,并使用由用户的移动决定的音量。A drawing interaction should get a persistent, looping sound with volume determined by the user's hand movement. 当用户的手仍在 loudest 时,它应为无提示。It should be silent when the user's hand is still and loudest when the hand is moving quickly.

语音交互Voice interactions

语音交互通常具有微妙的视觉元素。Voice interactions often have subtle visual elements. 使用声音来强化交互阶段。Use sounds to reinforce interaction stages. 您可能想要使用更多的声音,将它们与手势和直接操作声音区分开来。You may want to use more-tonal sounds to distinguish them from gesture and direct-manipulation sounds.

  • 语音命令 确认 使用有正负音。Use a positive-sounding tone for voice command confirmations. 升高的声音和主要的音乐间隔都是有效的。Rising tones and major musical intervals are effective.
  • 对于语音命令 失败,请使用较短的、不太积极的音调。Use a shorter, less-positive-sounding tone for voice command failures. 避免出现负面声音。Avoid negative sounds. 相反,请使用更 percussive 的中性声音来传达应用程序正在交互的活动。Instead, use a more percussive, neutral sound to communicate that the application is moving on from the interaction.
  • 如果你的应用程序具有唤醒字词,则在设备 开始侦听 时,请使用短暂的短暂音。If your application has a wake word, use a short, gentle tone when the device starts listening. 应用程序侦听时使用 细小循环声音。Use a subtle looping sound while the application is listening.

通知Notifications

通知通知应用程序状态更改和用户未启动的其他事件。Notifications signal application-state changes and other events the user didn't initiate. 状态更改可能包括进程完成、消息和电话呼叫。State changes can include process completions, messages, and phone calls.

在混合现实中,对象有时会移出用户的视图字段。In mixed reality, objects sometimes move out of the user's field of view. 使用依赖于对象类型和运动速度的 spatialized 声音进行配对移动 动画对象Pair moving animated objects with a spatialized sound that depends on the object type and speed of motion.

  • 它有助于在动画结束时播放 spatialized 声音,以通知用户对象的新位置。It helps to play a spatialized sound at the end of an animation to inform the user of the object's new position.
  • 对于逐步运动,移动过程中的 "whoosh" 声音有助于用户跟踪对象。For gradual movements, a "whoosh" sound during movement helps the user track the object.

消息通知 声音可能会反复听到,有时会迅速继续。Message notification sounds may be heard repeatedly, sometimes in quick succession. 这一点很重要,它们不会显得不上或太恶劣。It's important they don't stand out or sound harsh. 中间的正面色调声音有效。Mid-range positive tonal sounds are effective.

  • 传入呼叫声音对于手机铃声应该具有相似的质量。Incoming-call sounds should have similar qualities to a cell phone ringtone. 这些声音会循环播放音乐短语,直到用户应答呼叫。These sounds are looping musical phrases that play until the user answers the call.
  • 语音通信连接和断开连接应具有短暂的色调声音。Voice communication connection and disconnection should have a short, tonal sound. 连接声音应为正片,指示连接成功。The connection sound should be a positive tone to indicate a successful connection. 断开连接声音应该是一种中性声音,指示调用完成。The disconnection sound should be a neutral sound to indicate completion of the call.

处理 spatializationHandle spatialization

Spatialization 使用立体声耳机或扬声器将声音置于混合现实世界中。Spatialization uses stereo headphones or speakers to place sounds in the mixed-reality world.

哪些声音 spatializeWhich sounds to spatialize

当声音与具有空间位置的事件关联时,应 spatialized。A sound should be spatialized when it's associated with an event that has a spatial location. 这包括 UI、包含的 AI 声音和视觉指示器。This includes UI, embodied AI voices, and visual indicators.

Spatialize 用户界面 元素,可通过限制他们听到的立体声声音数量来帮助整理用户的 sonic "space"。Spatialize user interface elements to help declutter the user's sonic "space" by limiting the number of stereo sounds that they hear. 在 spatialized 音频反馈时,处理触摸、抓取和释放感觉更自然的交互。Manipulation interactions such as touching, grabbing, and releasing feel more natural when audio feedback is spatialized. 请考虑以下有关这些元素的距离衰减的信息。Consider the following information about distance attenuation for these elements.

Spatialize 直观的指示器 和三应的 AI 声音 ,以直观地通知用户这些东西超出了视图的范围。Spatialize visual indicators and embodied AI voices to intuitively inform users when these things are outside the field of view.

与此相反,应避免 spatialization 用于 FACELESS AI 声音 ,还应避免缺少定义良好的空间位置的其他元素。In contrast, avoid spatialization for faceless AI voices and other elements that lack a well-defined spatial location. 如果没有相关的视觉元素,Spatialization 可能会分散用户以为其无法找到的可视元素。Spatialization without a related visual element can distract users into thinking there's a visual element that they can't find.

Spatialization 有一定的 CPU 开销。Spatialization does come with some CPU cost. 许多应用程序最多可以同时播放两个声音。Many applications have at most two sounds playing simultaneously. 在这种情况下,spatialization 的成本可能会忽略不计。The cost of spatialization in that case is likely negligible. 可以使用 MRTK 帧速率监视器来判断添加 spatialization 的影响。You can use the MRTK frame rate monitor to judge the impact of adding spatialization.

何时以及如何应用基于距离的衰减When and how to apply distance-based attenuation

在现实生活中,远离远处的声音会更安静。In the physical world, sounds that are farther away are quieter. 音频引擎可以根据源距离对此衰减建模。Your audio engine can model this attenuation based on the source distance. 在传达相关信息时,请使用基于距离的衰减。Use distance-based attenuation when it communicates relevant information.

视觉对象指示器动画影像 和其他信息性声音的距离与用户相关。The distances to visual indicators, animated holograms, and other informative sounds are relevant to the user. 使用基于距离的衰减直观地提供提示。Use distance-based attenuation to intuitively provide cues.

调整每个源的衰减曲线,使其适应混合现实世界的空间大小。Adjust the attenuation curve for each source to fit the size of your mixed-reality world's spaces. 音频引擎的默认曲线通常适用于大 () 空间 kilometer。Your audio engine's default curve is often meant for large (up to half-kilometer) spaces.

强调按钮操作和其他交互的 渐进式阶段 的声音不应应用衰减。Sounds that reinforce the progressive stages of button actions and other interactions shouldn't get attenuation applied. 这些声音的加强效果比与按钮通信的重要性更重要。The reinforcing effects of these sounds are more important than communicating the distance to the button. 变体可能会分散注意力,尤其是在键盘上,很多按钮单击时可能会被连续听到。Variations can be distracting, especially with keyboards, when many button clicks may be heard in succession.

使用哪种 spatialization 技术Which spatialization technology to use

使用耳机或 HoloLens 扬声器,使用头相关的传输功能 (基于) 的 spatialization 技术。With headphones or the HoloLens speakers, use head-related transfer function (HRTF)-based spatialization technologies. 这些技术对现实世界内的声音的传播建模。These technologies model the sound propagation around the head in the physical world. 即使当某个声源位于一端时,声音仍会传播到远处的耳,同时出现一些衰减和延迟。Even when a sound source is on the far side of one's head, sound propagates to the distant ear with some attenuation and delay. 演讲者平移仅依赖于衰减,并在左侧的声音位于右侧,而另一种方式上应用总衰减。Speaker panning relies only on attenuation and applies total attenuation in the left ear when sounds are on the right side, and the other way around. 对于在一条耳中听力有障碍的侦听器,此方法可能不适合 "正常听觉" 侦听器并且无法访问。This technique can be uncomfortable for "normal hearing" listeners and inaccessible for listeners who have hearing impairment in one ear.

后续步骤Next steps