案例研究-在 RoboRaid 中使用空間音效Case study - Using spatial sound in RoboRaid

本文說明在針對 RoboRaid mixed reality first person 射擊建立音訊時,小組面臨的 Microsoft HoloLens 體驗挑戰。This article describes challenges the Microsoft HoloLens Experience Team faced while creating audio for the RoboRaid mixed-reality first-person shooter.

技術The tech

空間音效 是 Microsoft HoloLens 最令人興奮的功能之一,可讓使用者在沒有物件的情況下,察覺到他們的工作。Spatial sound is one of the most exciting features of Microsoft HoloLens, letting users perceive what's going on around them when objects aren't in the line of sight.

在 RoboRaid 中,最明顯且有效的空間音效用途,是提醒播放程式發生在其周邊視覺以外的地方。In RoboRaid, the most obvious and effective use of spatial sound is alerting the player to something happening outside of their peripheral vision. 例如,Breacher 可以輸入房間內掃描的任何牆。For example, the Breacher can enter from any of the scanned walls in the room. 如果您不想知道其進入的位置,可能會錯過。If you're not facing the location where it's entering, you might miss it. 若要向您發出此目睹的警示,您會聽到來自 Breacher 輸入位置的相異音訊,讓您知道您需要快速採取行動來停止。To alert you to this invasion, you'll hear a distinct bit of audio coming from where the Breacher is entering, which lets you know you need to act quickly to stop it.

在幕後Behind the scenes

為 HoloLens 應用程式建立空間音效的原因很新,而且獨特的問題可能很難解決,因為沒有任何過去的專案需要參考。Creating spatial sound for HoloLens apps is so new and unique that problems can be difficult to solve because there aren't any past projects to reference. 希望這些在進行 RoboRaid 時所面臨的音效挑戰範例,可協助您建立自己應用程式的音訊。Hopefully, these examples of the audio challenges we faced while making RoboRaid will help you as you create audio for your own apps.

留意 CPU 的負擔Be mindful of taxing the CPU

空間音效可能會要求 CPU。Spatial sound can be demanding on the CPU. 對於忙碌的 RoboRaid,在任何指定時間將空間音效實例的數目維持在八個底下是很重要的。For a busy experience like RoboRaid, it was crucial to keep the number of spatial sound instances to under eight at any given time. 通常,它就像設定不同音訊事件的實例限制一樣簡單。Usually, it was as easy as setting the limit of instances for different audio events. 在達到限制之後發生的任何實例都會被終止。Any instances that happen after the limit is reached are killed. 例如,無人機產生時,在任何指定時間,其 screams 會限制為三個實例。For example, when drones spawn, their screams are limited to three instances at any given time. 只考慮有四個無人機可以產生一次,三個 screams 很多,因為您的大腦無法追蹤許多類似發音的音訊事件。Considering only about four drones can spawn at once, three screams are plenty since there’s no way your brain can keep track of that many similar-sounding audio events. 這會釋出其他空間音效事件的資源,例如敵人爆炸或敵人準備進行。This freed up resources for other spatial sound events, like enemy explosions or enemies preparing to shoot.

獎勵成功的減減Rewarding a successful dodge

躲過技師修理是 RoboRaid 中最重要的遊戲機制之一,也是我們認為 HoloLens 體驗真正獨特的東西。The dodging mechanic is one of the most important gameplay mechanics in RoboRaid, and also something that we felt was truly unique to the HoloLens experience. 因此,我們想要讓玩家順利 dodges。As such, we wanted to make successful dodges very rewarding to the player. 我們 Doppler 「whizz 者」在開發過程中的初期很吸引人。We got the Doppler "whizz-by" to sound compelling fairly early on in the development. 一開始,我的計畫是使用迴圈,並使用音量、音調和篩選來即時操作它。Initially, my plan was to use a loop and manipulate it in real-time using volume, pitch, and filter. 這項作業的執行非常複雜。The implementation for this was going to be very elaborate. 在認可資源以建立這項功能之前,我們使用了 Doppler 效果內建的資產建立了一個便宜的原型,以找出它的感覺。Before committing resources to build this we created a cheap prototype using an asset with the Doppler effect baked in just to find out how it felt. 我們的優秀開發人員讓此 whizz 的資產在 projectile 通過播放程式的 ear 之前,會播放正好0.7 秒,而結果則令人驚奇!Our talented dev made it so that this whizz-by asset would play back exactly 0.7 second before the projectile will have passed by the player’s ear and the results felt amazing! 不用說,我們拋棄了更複雜的方案,並實作為原型。Needless to say, we ditched the more complex solution and implemented the prototype.

(如需有關使用內建 Doppler 效果建立音訊資產的詳細資訊,請參閱 2 分鐘內的 100 Whooshes。 ) (For more information about creating an audio asset with the Doppler effect built in, see 100 Whooshes in 2 Minutes.)
成功躲過敵人的 projectile 利用符合 whizz 的音效來獎勵玩家。Successfully dodging an enemy's projectile rewards the player with a satisfying whizz-by sound.

拋棄失效音效Ditching ineffective sounds

一開始,我們想要在播放程式成功 dodged 敵人 projectile 後播放爆炸音效,但我們決定基於數個原因來揚棄。Originally, we had wanted to play an explosion sound behind the player once they’ve successfully dodged the enemy projectile, but we decided to ditch this for several reasons. 首先,它並不像我們用來進行減至 whizz 的 SFX 一樣有效率。First, it didn’t feel as effective as the whizz-by SFX we used for the dodge. 當 projectile 碰到您背後的牆時,遊戲中的其他東西可能會遮罩該音效。By the time the projectile hits a wall behind you, something else would have happened in the game that would mask that sound. 其次,我們的樓層沒有任何衝突,因此當 projectile 碰到地面而非牆壁時,我們就無法讓您播放激增。Second, we didn’t have collision on the floor, so we couldn’t get the explosion to play when the projectile hit the floor instead of the walls. 最後,空間音效的 CPU 成本也是如此。And finally, there was the CPU cost of spatial sound. Scorpion 敵人 (可以在牆內進行編目的資訊,) 有一項特殊攻擊,寄大約八炮彈。The Elite Scorpion enemy (one that can crawl inside the wall) has a special attack that shoots about eight projectiles. 這並不只是造成混合的巨大改變,也引進了一種劈啪聲,因為它的 CPU 太困難。Not only did that make a huge mess in the mix, it also introduced awful crackling because it was hitting the CPU too hard.

傳達命中Communicating a hit

我們在 HoloLens 上遇到一個有趣的問題,就是要有效傳達玩家的通訊難度。An interesting issue we ran into on the HoloLens was how difficult it was to effectively communicate that a player had been hit. 混合現實體驗的成功之處,就是您覺得案例正在發生。What makes a mixed reality experience successful is the feeling that the story is happening to you. 這表示您必須在自己的客廳中,相信您正在對抗一間的機器人目睹。That means you have to believe YOU are fighting an alien robot invasion in your own living room.

當玩家遇到問題時,他們顯然不會覺得任何東西,因此我們必須找出一種方法來說服玩家發生錯誤。Players obviously won't feel anything when they get hit, so we had to find a way to convince the player that something bad happened to them. 在傳統遊戲中,您可能會看到一個動畫,讓您知道您的字元已被叫用,或螢幕可能會閃爍紅色,而您的字元可能有點 grunt。In conventional games, you might see an animation that lets you know your character has taken a hit, or the screen might flash red and your character might grunt a little. 由於這些類型的提示在混合的現實體驗中無法運作,因此我們決定將視覺提示與放大的音效合併,以指出您已損毀。Since these types of cues don't work in a mixed reality experience, we decided to combine the visual cue with an exaggerated sound that indicates you've taken damage. 我建立了一個很大的音效,讓它在混合 ducked 所有專案的情況下變得很明顯。I created a big sound, and made it so prominent in the mix that it ducked everything down. 然後,為了讓它更有意義,我們新增了一個短暫的警告音效,就像是要接收到一個Then, to make it stand out even more, we added a short warning sound as if a nuclear sub was sinking.
當播放程式在 RoboRaid 中叫用時,會看到視覺提示,但也會收到放大的音訊提示,告知他們已損毀。When a player gets hit in RoboRaid, they see a visual cue, but also get an exaggerated audio cue that tells them they've taken damage.

從小型喇叭取得大音效Getting big sound from small speakers

HoloLens 喇叭很小且符合裝置的需求,因此您不能預期會聽到太多的低終端。HoloLens speakers are small and light to suit the needs of the device, so you can’t expect to hear too much low-end. 類似于為智慧型手機或掌上型遊戲裝置進行開發,音效設計工具和作曲家必須留意音訊的頻率內容。Similar to developing for smart phones or handheld gaming devices, sound designers and composers have to be mindful of the frequency content of their audio. 我總是設計音效或以完整頻率範圍撰寫音樂,因為戴耳機是使用者的選項。I always design sounds or write music with full frequency range because wearing headphones is an option for the users. 不過,為了確保與 HoloLens 喇叭的相容性,我偶爾會在執行的任何白苗文的主伺服器上放置 EQ,藉以執行測試。However, to ensure compatibility with HoloLens speakers, I run a test occasionally by putting an EQ in the master of any DAW I happen to be working in. EQ 設定是由大約 600 Hz 到 700 Hz 的高通過篩選器所組成 () 和低傳遞篩選器,大約是1萬 (陡) 。The EQ setting consists of a high-pass filter around 600 Hz to 700 Hz (not too steep) and low-pass filter at around 10K (steep). 這應該能讓您大致瞭解聲音在裝置上的播放方式。That should give you an approximate idea of how your sounds will play back on the device.

如果您想要使用低音來提供您的音樂中的弦變更,您可能會發現當您套用此 EQ 設定時,您的音樂完全失去了根目錄的意義。If you're relying on bass to give the sense of chord changing in your music, you may find that your music completely loses the sense of root when you apply this EQ setting. 為了解決這個情況,我將另一層新增至低音,也就是一個 octave (有一些豐富的諧波) ,並混用它來瞭解 root 的意義。To remedy this, I added another layer to the bass that is one octave higher (with some rich harmonics) and mixed it to get the sense of root back. 有時候,使用失真來 amp 諧波,會在上半部提供足夠的頻率內容,讓我們的大腦覺得它下有一些東西。Sometimes using distortion to amp up the harmonics will give enough frequency content in the upper range to make our brain think that there’s something underneath it. 這適用于 SFX、爆炸或音效等的特殊時刻(例如老闆的超級攻擊)。This is true for SFX like impacts, explosions, or sounds for special moments, such as a boss' super attacks. 您真的無法依賴低端來讓玩家有影響或權數的意義。You really can’t rely on the low-end to give the player a sense of impact or weight. 與音樂一樣,使用失真讓某些 crunch 絕對有説明。As with music, using distortion to give some crunch definitely helped.

讓您的音訊提示更醒目Making your audio cues stand out

當然,小組中的每個人都想要 bombastic 音樂、大聲主張和奇怪的爆炸;但是他們也希望能夠聽到 voiceover 或任何其他遊戲關鍵的音訊提示。Naturally, everyone on the team wanted bombastic music, loud guns, and crazy explosions; but they also wanted to be able to hear voiceover or any other game-critical audio cues.

在具有完整頻率範圍的主控台遊戲中,您有更多選項可以根據音效的重要性來劃分頻率。On a console game with full range of frequency, you have more options to divide frequencies up depending on the importance of the sound. 對 RoboRaid 來說,我的頻率範圍有限,我可以從音效曲線。For RoboRaid, I was limited in the number of ranges of frequencies I could curve out from sounds. 如果您在範圍較高的範圍內使用低通過篩選和曲線,您將不會有任何剩餘的聲音,因為沒有太多的低端。If you use low-pass filter and curve out too much from the higher end of the spectrum, you won’t have anything left on the sound because there’s not much low-end.

若要讓 RoboRaid 音效盡可能地在裝置上變得很大,我們必須降低整個體驗的動態範圍,並藉由為不同類型的音效建立明確的重要性階層,來廣泛使用回避。To make RoboRaid sound as big as it can on the device, we had to lower the dynamic range of the whole experience and made extensive use of ducking by creating a clear hierarchy of importance for different types of sounds. 我根據重要性將回避從-2 dB 設定為-6 dB。I set the ducking from -2 dB to -6 dB depending on the importance. 我個人並不喜歡遊戲中的明顯回避,所以我花了很多時間來微調淡入/退出時間以及音量衰減量。I personally don’t like obvious ducking in games, so I spent a lot of time tuning the fade in/out timing and the amount of volume attenuation. 我們針對空間音效、非空間音效、VO 和幹匯流排設定個別的匯流排,而不會有音樂的回音。We set up separate busses for spatial sound, non-spatial sound, VO, and dry bus without reverb for music. 然後,我們建立了高優先順序、關鍵和非關鍵的匯流排,讓資產設定為進入適當的匯流排。Then, we created high priority, critical, and non-critical busses so the assets were set up to go to their appropriate busses.

我希望音訊專業人員在處理自己的應用程式時,會有很多有趣和驚喜,因為我在 RoboRaid 上工作。I hope audio professionals out there will have as much fun and excitement working on their own apps as I did working on RoboRaid. 我無法再看看 (並聽聽! ) Microsoft 外部的員工會在 HoloLens 上提出哪些內容。I can’t wait to see (and hear!) what the talented folks outside Microsoft will come up with for HoloLens.

親自完成Do it yourself

我發現要讓特定事件 ((例如,爆炸) 音效「更大」)(像是填滿房間)的一個訣竅,就是為空間音效建立 mono 資產,並混用2D 身歷聲資產,以3D 播放。One trick I discovered to make certain events (such as explosions) sound "bigger"—like they're filling up the room—was to create a mono asset for the spatial sound and mix it with a 2D stereo asset, to be played back in 3D. 它的確需要進行一些調整,因為身歷聲內容中有太多資訊會減少 mono 資產的方向。It does take some tuning, since having too much information in the stereo content will lessen the directionality of the mono assets. 不過,取得餘額將會產生很大的音效,讓玩家以正確方向來轉換其標頭。However, getting the balance right will result in huge sounds that will get players to turn their heads in the right direction.

您可以使用下列音訊資產自行試用音效:You can try out big sounds yourself using the audio assets below:

案例 1Scenario 1

  1. 下載 roboraid_enemy_explo_mono .wav ,並設定為透過空間音效播放,並將其指派給事件。Download roboraid_enemy_explo_mono.wav and set to play back through spatial sound and assign it to an event.
  2. 下載 roboraid_enemy_explo_stereo .wav ,並將其設定為在2d 身歷聲中播放,並指派給上述的相同事件。Download roboraid_enemy_explo_stereo.wav and set to play back in 2D stereo and assign to the same event as above. 因為這些資產會正規化為 Unity,所以 attenuate 兩個資產的數量,使其不會被裁剪。Because these assets are normalized to Unity, attenuate volume of both assets so that it doesn’t clip.
  3. 同時玩這兩個音效。Play both sounds together. 移動您的頭部,以感受空間的音效。Move your head around to feel how spatial it sounds.

案例 2Scenario 2

  1. 下載 roboraid_enemy_explo_summed .wav ,並設定為透過空間音效播放並指派給事件。Download roboraid_enemy_explo_summed.wav and set to play back through spatial sound and assign to an event.
  2. 自行播放此資產,然後將其與案例1中的事件進行比較。Play this asset by itself then compare it to the event from Scenario 1.
  3. 試用 mono 和身歷聲檔案的不同餘額。Try different balance of mono and stereo files.

請參閱See also