案例研究-在 RoboRaid 中使用空间音效Case study - Using spatial sound in RoboRaid

本文介绍 Microsoft HoloLens 体验团队在为 RoboRaid mixed reality 第一人称射击创建音频时所面临的挑战。This article describes challenges the Microsoft HoloLens Experience Team faced while creating audio for the RoboRaid mixed-reality first-person shooter.

技术人员The tech

空间音效 是 Microsoft HoloLens 最令人兴奋的功能之一,让用户感觉到当对象不在视觉上时,用户会感觉到它们的进展。Spatial sound is one of the most exciting features of Microsoft HoloLens, letting users perceive what's going on around them when objects aren't in the line of sight.

在 RoboRaid 中,空间音质最明显且最有效的使用情况是提醒播放机在其外围视力之外发生的事情。In RoboRaid, the most obvious and effective use of spatial sound is alerting the player to something happening outside of their peripheral vision. 例如,Breacher 可以输入房间中的任何扫描壁。For example, the Breacher can enter from any of the scanned walls in the room. 如果你不是面对其进入位置,可能会错过。If you're not facing the location where it's entering, you might miss it. 为了提醒您此 invasion,您将收到 Breacher 进入的位置的不同位音频,这让您知道您需要快速地停止它。To alert you to this invasion, you'll hear a distinct bit of audio coming from where the Breacher is entering, which lets you know you need to act quickly to stop it.

幕后Behind the scenes

为 HoloLens 应用创建空间音效是如此新且独特的问题可能难以解决,因为没有任何以前要引用的项目。Creating spatial sound for HoloLens apps is so new and unique that problems can be difficult to solve because there aren't any past projects to reference. 希望在创建 RoboRaid 时,我们所面临的音频挑战的示例可帮助你为自己的应用创建音频。Hopefully, these examples of the audio challenges we faced while making RoboRaid will help you as you create audio for your own apps.

注意 CPU 的负担Be mindful of taxing the CPU

在 CPU 上可以要求空间音质。Spatial sound can be demanding on the CPU. 对于繁忙的体验(如 RoboRaid),在任何给定时间,将空间声音实例的数量保持在八位是至关重要的。For a busy experience like RoboRaid, it was crucial to keep the number of spatial sound instances to under eight at any given time. 通常,与为不同的音频事件设置实例限制一样简单。Usually, it was as easy as setting the limit of instances for different audio events. 达到限制后发生的任何实例都将被终止。Any instances that happen after the limit is reached are killed. 例如,在无人机生成时,在任何给定时间,其 screams 限制为三个实例。For example, when drones spawn, their screams are limited to three instances at any given time. 如果只考虑四个无人机可以一次生成,三个 screams 都非常多,因为您的大脑无法跟踪很多发音相似的音频事件。Considering only about four drones can spawn at once, three screams are plenty since there’s no way your brain can keep track of that many similar-sounding audio events. 这会释放其他空间声音事件的资源,例如敌人爆炸或敌人准备进行拍摄。This freed up resources for other spatial sound events, like enemy explosions or enemies preparing to shoot.

获得成功的减减Rewarding a successful dodge

Dodging mechanic 是 RoboRaid 中最重要的游戏机制之一,也是我们认为在 HoloLens 体验中真正唯一的东西。The dodging mechanic is one of the most important gameplay mechanics in RoboRaid, and also something that we felt was truly unique to the HoloLens experience. 这种情况下,我们想让 dodges 对播放机取得成功。As such, we wanted to make successful dodges very rewarding to the player. 在开发中,我们的 Doppler "whizz" 听起来非常引人注目。We got the Doppler "whizz-by" to sound compelling fairly early on in the development. 最初,我的计划是使用循环并使用卷、间距和筛选器对其进行实时操作。Initially, my plan was to use a loop and manipulate it in real-time using volume, pitch, and filter. 此操作的实现非常复杂。The implementation for this was going to be very elaborate. 在提交资源以生成此操作之前,我们创建了一个经济的原型,其中使用的资产具有 Doppler 效果融入,只是为了查明其感觉。Before committing resources to build this we created a cheap prototype using an asset with the Doppler effect baked in just to find out how it felt. 我们的共同开发人员使得这一 whizz 资产将在 projectile 通过玩家的耳之前0.7 播放,而结果却很惊人!Our talented dev made it so that this whizz-by asset would play back exactly 0.7 second before the projectile will have passed by the player’s ear and the results felt amazing! 不用说,我们借助了更复杂的解决方案并实现了原型。Needless to say, we ditched the more complex solution and implemented the prototype.

(有关使用内置的 Doppler 效果创建音频资产的详细信息,请参阅 100 Whooshes in 2 分钟。 ) (For more information about creating an audio asset with the Doppler effect built in, see 100 Whooshes in 2 Minutes.)
成功 dodging 敌人的 projectile 通过一种符合的 whizz 来奖励玩家。Successfully dodging an enemy's projectile rewards the player with a satisfying whizz-by sound.

抛弃无效声音Ditching ineffective sounds

最初,我们希望在成功 dodged 敌人 projectile 后,在玩家后面播放一声爆炸形声,但我们决定出于多种原因而 ditch。Originally, we had wanted to play an explosion sound behind the player once they’ve successfully dodged the enemy projectile, but we decided to ditch this for several reasons. 首先,这并不像我们用于减减的 whizz。First, it didn’t feel as effective as the whizz-by SFX we used for the dodge. 当 projectile 碰到您的墙后,就会在游戏中出现其他问题。By the time the projectile hits a wall behind you, something else would have happened in the game that would mask that sound. 其次,我们在地面上没有冲突,因此当 projectile 到达地面而不是墙壁时,我们无法进行分解。Second, we didn’t have collision on the floor, so we couldn’t get the explosion to play when the projectile hit the floor instead of the walls. 最后,空间音质会出现 CPU 开销。And finally, there was the CPU cost of spatial sound. 精英 Scorpion 敌人 (可以在墙壁内进行爬网的炮弹) 有一种特殊的攻击桃约八。The Elite Scorpion enemy (one that can crawl inside the wall) has a special attack that shoots about eight projectiles. 这不仅会使混合中出现巨大的混乱,而且还引入了一噼啪的噼啪声,因为它会导致 CPU 太难。Not only did that make a huge mess in the mix, it also introduced awful crackling because it was hitting the CPU too hard.

通信命中Communicating a hit

我们在 HoloLens 上遇到了一个有趣的问题,即很难有效地传达播放机。An interesting issue we ran into on the HoloLens was how difficult it was to effectively communicate that a player had been hit. 这会使混合现实体验成功,这就是你觉得故事正在进行。What makes a mixed reality experience successful is the feeling that the story is happening to you. 这意味着你必须相信你要在自己的客厅里反击一 invasion 外部的机器人。That means you have to believe YOU are fighting an alien robot invasion in your own living room.

当玩家遇到问题时,他们很显然不会感觉任何问题,因此我们必须找到一种方法来说服玩家发生错误。Players obviously won't feel anything when they get hit, so we had to find a way to convince the player that something bad happened to them. 在传统游戏中,你可能会看到一个动画,它可以让你知道你的人物是否发生了点击,或者屏幕可能会以红色闪烁,你的人物可能 grunt。In conventional games, you might see an animation that lets you know your character has taken a hit, or the screen might flash red and your character might grunt a little. 由于这些类型的提示在混合现实体验中不起作用,因此我们决定将视觉提示与一个放大的声音组合在一起,以指示您已遭到破坏。Since these types of cues don't work in a mixed reality experience, we decided to combine the visual cue with an exaggerated sound that indicates you've taken damage. 我创建了一个很好的声音,并使其突出,使其 ducked。I created a big sound, and made it so prominent in the mix that it ducked everything down. 那么,为了使其更进一步,我们添加了一条短暂的警告音,就像是一个核 sub。Then, to make it stand out even more, we added a short warning sound as if a nuclear sub was sinking.
当播放机在 RoboRaid 中命中时,它们会看到视觉提示,但也会获得一个放大的音频提示,告知他们已损坏。When a player gets hit in RoboRaid, they see a visual cue, but also get an exaggerated audio cue that tells them they've taken damage.

从小扬声器获得大声音Getting big sound from small speakers

HoloLens 扬声器很小,并且非常适合于设备的需求,因此您无法听到太多的低端。HoloLens speakers are small and light to suit the needs of the device, so you can’t expect to hear too much low-end. 与开发智能手机或手持游戏设备类似,声音设计人员和作曲者必须注意其音频的频率内容。Similar to developing for smart phones or handheld gaming devices, sound designers and composers have to be mindful of the frequency content of their audio. 我始终设计声音或使用全频率范围写入音乐,因为戴耳机是用户的一个选项。I always design sounds or write music with full frequency range because wearing headphones is an option for the users. 但是,为了确保与 HoloLens 扬声器的兼容性,我偶尔运行测试,方法是在发生的任何 DAW 的主节点中放置一个 EQ。However, to ensure compatibility with HoloLens speakers, I run a test occasionally by putting an EQ in the master of any DAW I happen to be working in. EQ 设置由围绕 600 Hz 的高通滤波器到 700 Hz, (不是太陡) 和低刀筛选器大约为 10K (陡) 。The EQ setting consists of a high-pass filter around 600 Hz to 700 Hz (not too steep) and low-pass filter at around 10K (steep). 这应该会使您大致了解一下声音如何播放到设备上。That should give you an approximate idea of how your sounds will play back on the device.

如果你要依赖于低音来在音乐中进行更改,则在应用此 EQ 设置时,你可能会发现音乐完全丢失了 root。If you're relying on bass to give the sense of chord changing in your music, you may find that your music completely loses the sense of root when you apply this EQ setting. 若要解决此情况,我向低音添加了另一层,这是一个 octave 更高的 (,具有一些丰富的谐波) ,并将其混合在一起以获得 root。To remedy this, I added another layer to the bass that is one octave higher (with some rich harmonics) and mixed it to get the sense of root back. 有时,使用扭曲来使谐波会在上部范围内提供足够的频率内容,以使大脑认为它下面有一些内容。Sometimes using distortion to amp up the harmonics will give enough frequency content in the upper range to make our brain think that there’s something underneath it. 这种情况适用于类似于影响、爆炸或声音等一段时间的,例如老板的超级攻击。This is true for SFX like impacts, explosions, or sounds for special moments, such as a boss' super attacks. 你确实不能依赖低端来使玩家了解影响或权重。You really can’t rely on the low-end to give the player a sense of impact or weight. 与音乐一样,使用扭曲来为某些捕获提供一定的帮助。As with music, using distortion to give some crunch definitely helped.

使您的音频提示突出Making your audio cues stand out

当然,团队中的每个人都希望 bombastic 音乐、枪支和古怪爆炸;但他们还希望能够听到 voiceover 或任何其他游戏关键的音频提示。Naturally, everyone on the team wanted bombastic music, loud guns, and crazy explosions; but they also wanted to be able to hear voiceover or any other game-critical audio cues.

在具有各种频率的控制台游戏上,根据声音的重要性,你可以使用更多的选项来划分频率。On a console game with full range of frequency, you have more options to divide frequencies up depending on the importance of the sound. 对于 RoboRaid,我限制了从声音弯曲的频率范围。For RoboRaid, I was limited in the number of ranges of frequencies I could curve out from sounds. 如果使用低刀筛选器并使其在较高的范围内弯曲,则不会有任何内容出现在声音上,因为没有太多的低端。If you use low-pass filter and curve out too much from the higher end of the spectrum, you won’t have anything left on the sound because there’s not much low-end.

为了使 RoboRaid 的声音与设备上的声音非常大,我们必须通过为不同类型的声音创建清晰的重要性层次结构,来降低整个体验的动态范围并广泛使用放掉。To make RoboRaid sound as big as it can on the device, we had to lower the dynamic range of the whole experience and made extensive use of ducking by creating a clear hierarchy of importance for different types of sounds. 根据重要性,将放掉从-2 dB 设置为-6 dB。I set the ducking from -2 dB to -6 dB depending on the importance. 我个人不喜欢在游戏中显而易见的放掉,所以我花了很多时间来优化淡入/淡出时间和体积衰减量。I personally don’t like obvious ducking in games, so I spent a lot of time tuning the fade in/out timing and the amount of volume attenuation. 我们为空间音效、非空间音效、VO 和干燥总线设置了单独的总线,无回音用于音乐。We set up separate busses for spatial sound, non-spatial sound, VO, and dry bus without reverb for music. 接下来,我们创建了高优先级、关键且非关键的总线,因此这些资产已设置为可用于适当的总线。Then, we created high priority, critical, and non-critical busses so the assets were set up to go to their appropriate busses.

我希望音频专业人员在处理 RoboRaid 时,会在其自己的应用上工作时有很多乐趣和惊喜。I hope audio professionals out there will have as much fun and excitement working on their own apps as I did working on RoboRaid. 我看不到 (,听不到! ) Microsoft 在为 HoloLens 提供的工作人员。I can’t wait to see (and hear!) what the talented folks outside Microsoft will come up with for HoloLens.

自制Do it yourself

我发现了某些事件 (如爆炸) "更大" 的声音,就像是在填满房间,就是为空间音效创建单声道资产,并将其与2D 立体声资产混合,以三维方式播放。One trick I discovered to make certain events (such as explosions) sound "bigger"—like they're filling up the room—was to create a mono asset for the spatial sound and mix it with a 2D stereo asset, to be played back in 3D. 它确实会进行一些优化,因为立体声内容中的信息过多会降低 mono 资产的方向。It does take some tuning, since having too much information in the stereo content will lessen the directionality of the mono assets. 但是,若要获得平衡,将会产生非常大的声音,让玩家按正确的方向。However, getting the balance right will result in huge sounds that will get players to turn their heads in the right direction.

您可以使用下面的音频资产自行尝试大声音:You can try out big sounds yourself using the audio assets below:

方案 1Scenario 1

  1. 下载 roboraid_enemy_explo_mono .wav ,并将其设置为通过空间音效播放,并将其分配给某个事件。Download roboraid_enemy_explo_mono.wav and set to play back through spatial sound and assign it to an event.
  2. 下载 roboraid_enemy_explo_stereo .wav ,并将其设置为在2d 立体声中播放,并将其分配给上述相同的事件。Download roboraid_enemy_explo_stereo.wav and set to play back in 2D stereo and assign to the same event as above. 由于这些资产被规范化为 Unity,因此,这两个资产的衰减量使其不会被剪裁。Because these assets are normalized to Unity, attenuate volume of both assets so that it doesn’t clip.
  3. 同时播放两个声音。Play both sounds together. 四处移动您的头,看看它的工作方式。Move your head around to feel how spatial it sounds.

方案 2Scenario 2

  1. 下载 roboraid_enemy_explo_summed .wav ,并将其设置为通过空间音效播放并分配给某个事件。Download roboraid_enemy_explo_summed.wav and set to play back through spatial sound and assign to an event.
  2. 单独播放此资产,然后将其与方案1中的事件进行比较。Play this asset by itself then compare it to the event from Scenario 1.
  3. 尝试对 mono 和立体声文件进行不同的平衡。Try different balance of mono and stereo files.

另请参阅See also