案例研究: HoloTour 的空间音效设计Case study: Spatial sound design for HoloTour

全景视频和全息风景只是用于沉浸式 Microsoft HoloLens 教程的公式的一部分。Panoramic videos and holographic scenery are only part of the formula for an immersive Microsoft HoloLens tour. 本文介绍如何使用声音来使你真正进入每个 HoloTour 位置。This article describes how sound was used to make you feel like you're actually in each HoloTour location.

技术人员The tech

你在 HoloTour 中看到的精美图像和全息场景只是可信混合现实体验的一部分。The beautiful imagery and holographic scenes that you see in HoloTour are only one part of a believable mixed-reality experience. 尽管全息影像仅出现在用户面前,但 HoloLens 可以从所有方向提供 空间音效 ,这提供了更完整的传感器体验。While holograms only appear in front of a user, HoloLens can deliver spatial sound from all directions, which provides a more-complete sensory experience.

空间音效提供指示用户应关闭的方向,或让用户知道在其空间内有更多的全息影像的提示。Spatial sound provides cues to indicate a direction the user should turn or to let the user know there are more holograms to see within their space. 我们还可以将声音直接附加到全息图,并连续更新来自用户的全息图的方向和距离。We can also attach a sound directly to a hologram and continually update the direction and distance that the hologram is from the user. 此方法使其看起来像是直接来自该对象。This technique makes it seem as if the sound is coming directly from that object.

对于 HoloTour,我们想要充分利用 HoloLens 的空间音效功能,因此我们创建了一个与视频同步的360度环境环境,用于显示特定位置的 sonic 亮点。For HoloTour, we wanted to take advantage of the spatial sound capabilities of HoloLens, so we created a 360-degree ambient environment that's synchronized with the video to reveal the sonic highlights of specific locations.

幕后Behind the scenes

我们创建了两个不同位置的 HoloTour 体验:罗马和 Machu Picchu。We created HoloTour experiences of two different locations: Rome and Machu Picchu. 为了使这些教程外观真实且引人注目,我们想要从我们 filmed 的位置捕获音频,而不是使用一般声音。To make these tours feel authentic and compelling, we wanted to capture audio from the locations where we filmed instead of using generic sounds.

捕获音频Capturing the audio

在我们 关于捕获 HoloTour 视觉内容的案例研究中,我们讨论了照相机工作机组的设计。In our case study about capturing the visual content for HoloTour, we talk about the design of our camera rig. 它包括3D 打印架中的14个 GoPro 摄像机,设计用于适合该架。It consisted of 14 GoPro cameras in a 3D-printed housing that was designed to fit the tripod. 为了捕获音频,我们在相机下面添加了一个四重麦克风阵列。To capture audio, we added a quad-microphone array below the cameras. 声音将进入紧凑的四通道记录单元。The sound feeds into a compact four-channel recording unit at the base of the tripod. 我们选择的麦克风的确不错,但要避免干扰相机。We chose microphones that did well but were small enough to avoid interfering with the cameras.

自定义相机和麦克风装置Custom-made camera and microphone rig
自定义相机和麦克风装置Custom camera and microphone rig

此安装程序通过四个方向捕获声音。This setup captures sound in four directions. 我们记录了足够的信息来重新创建空间音效的3D 听觉全景,稍后我们可以同步到360度视频。We recorded enough information to re-create a 3D aural panorama of spatial sound, which we could later synchronize to the 360-degree video.

照相机阵列音频的一大挑战是,您受到的是相机声音,如 sirens、飞机或高释放。One challenge of the camera-array audio is that you're at the mercy of off-camera sounds, such as sirens, airplanes, or high winds. 为了确保我们具有我们所需的所有声音元素,我们使用了立体声和单声道移动录像机来捕获每个位置的特定兴趣点上的异步、环境声音。To make sure we had all the sound elements we needed, we used stereo and mono mobile recorders to capture asynchronous, ambient sound at specific points of interest in each location. 这些记录为声音设计器呈现了清晰的内容,以便在后期生产中增加兴趣并提高方向性。These recordings give the sound designer clean content to add interest and improve directionality in post-production.

每个捕获日生成很多文件。Every capture day generates many files. 因此,开发系统以跟踪与特定位置或相机拍摄相对应的文件非常重要。So it was important to develop a system to track which files correspond to a particular location or camera shot. 我们的记录单位设置为按日期自动命名文件并 "take" 号。Our recording unit was set up to automatically name files by date and "take" number. 在每天结束时,我们将文件备份到外部驱动器。We backed up files to external drives at the end of each day. 我们还发现,口头方式传达一开始录制音频录音非常重要。We also found it important to verbally slate the beginning of the audio recordings. 此预防措施允许在出现文件名问题时轻松地标识内容的上下文标识。This precaution allows easy contextual identification of the content in case file-name problems occur. 由于视频和音频记录为单独的媒体并且必须在生产后进行同步,因此,直观地显示照相机工作区捕获也很重要。It was also important to visually slate the camera rig capture, because the video and audio were recorded as separate media and had to be synchronized during post-production.

编辑音频Editing the audio

在捕获行程完成后,第一步是收集定向和沉浸式听觉体验的第一步是查看某个位置的所有捕获音频。Back at the studio after the capture trip, the first step in assembling a directional and immersive aural experience is to review all the captured audio for a location. 我们选取最好的做法,并确定可在集成期间应用的突出显示内容。We pick the best takes and identify highlights that can be applied during integration. 然后,将编辑并清理音频。The audio is then edited and cleaned up. 例如,可能会在同一捕获会话中用更安静的环境音频来替换持续一秒钟或重复几次的车载。For example, a car horn blast that lasts a second or so and repeats a few times might be replaced with quieter ambient audio from the same capture session.

设置位置的视频编辑后,声音设计器可以同步相应的音频。After the video edit for a location is set, the sound designer can synchronize the corresponding audio. 此时,我们将同时评估相机设备和移动版声音捕获,以确定哪些元素会构建最佳的沉浸式音频场景。At this point, we assess both the camera-rig and mobile sound captures to decide which elements would build the best immersive audio scene. 我们发现将所有声音元素放入音频编辑器并构建快速线性模拟,以试验不同的组合创意,这非常有用。We found it useful to put all the sound elements into an audio editor and build quick linear mock-ups to experiment with different mix ideas. 此步骤使我们能够更好地了解何时构建实际的 HoloTour 场景。This step gave us better-formed ideas for when it came time to build the actual HoloTour scenes.

组合场景Assembling the scene

构建三维环境场景的第一步是创建一个一般背景环境循环声音平台,它将支持场景中的其他功能和交互式声音元素。The first step to building a 3D ambient scene is to create a bed of general background ambient looping sounds that will support other features and interactive sound elements in a scene. 我们采用一种全面的方法来实现,具体取决于任何特定场景的设计标准。We take a holistic approach to implementation as determined by the design criteria for any particular scene. 某些场景可能使用同步的相机捕获进行索引。Some scenes might index toward using the synchronized camera capture. 更多的 "迷人" 时间可能需要特选的方法,该方法依赖于单独放置的声音、交互式元素和音乐。More "cinematic" moments might require a curated approach that relies on discretely placed sounds, interactive elements, and music.

当我们在相机捕获音频上编制索引时,我们将对应的空间启用了音量良好的环境音频发射器置于摄像机的方向。When we indexed on the camera-capture audio, we placed spatial sound-enabled ambient audio emitters that corresponded to the directional orientation of the cameras. "北方相机" 视图从 "中" 麦克风播放音频,对于其他关键方向也是如此。The north camera view plays audio from the north microphone, and likewise for the other cardinal directions. 这些发射器处于世界锁定状态,这意味着当用户开启打印头时,声音会改变。These emitters are world-locked, which means the sound changes when users turn their heads. 此方法可以有效地为在该位置的声音建模。This technique effectively models the sound of standing at that location. 倾听 Piazza Navona 或 Pantheon,了解使用良好的相机捕获音频的场景示例。Listen to Piazza Navona or The Pantheon for examples of scenes that use a good mix of camera-captured audio.

使用其他方法时,我们有时会播放立体声环境,其中包含位于场景周围的空间音效发射器。In a different approach, we sometimes play looping stereo ambience with spatial sound emitters that are placed around the scene. 这些发射器播放随机卷、间距和触发频率的一次性声音。These emitters play one-off sounds of randomized volume, pitch, and trigger frequency. 此方法创建了具有更强方向性的环境。This technique creates ambience that has an enhanced sense of directionality. 例如,在 Aguas Alienates 中,可以听到全景的每个象限如何具有特定的发射器,突出显示了地理位置的特定区域,但一起工作以创建总体沉浸式环境。In Aguas Alienates, for example, you can hear how each quadrant of the panorama has specific emitters that highlight specific areas of the geography but work together to create an overall immersive ambience.

提示和技巧Tips and tricks

还可以通过其他方式突出显示方向性并改进浸入式,充分利用 HoloLens 的空间音效功能。There are other ways to highlight directionality and improve immersion to make full use of the spatial sound capabilities of HoloLens. 我们在此处提供了一个列表。We've provided a list here. 下次尝试 HoloTour 时,请侦听这些效果。Listen for these effects the next time that you try HoloTour.

  • 查找目标: 当你查看全息帧的特定对象或区域时,将触发这些声音。Look targets: These sounds are triggered when you look at a specific object or area of the holographic frame. 例如,看一下罗马的 Piazza Navona 中的街道端咖啡馆,以更细微地触发繁忙的餐厅。For example, look toward the street-side café in Rome's Piazza Navona to subtly trigger busy-restaurant sounds.
  • 本地愿景: 尽管 HoloTour 的旅程包含某些 "节拍",其中的浏览指南通过全息影像进行深入探讨。Local vision: The journey though HoloTour contains certain "beats" where your tour guide, aided by holograms, explores a topic in-depth. 例如,作为 Pantheon 的外观,以显示 oculus,reverberating 音频作为来自 Pantheon 内部的3D 发射器,鼓励用户浏览内部。For instance, as the façade of the Pantheon dissolves to reveal the oculus, reverberating audio that was placed as a 3D emitter from inside the Pantheon encourages the user to explore the interior.
  • 增强方向: 在许多场景中,我们以各种方式将声音置于多方向中。Enhanced directionality: Within many scenes, we placed sounds in various ways to add to directionality. 例如,在 Pantheon 场景中,fountain 的声音作为单独的发射器放置在靠近用户的位置,用户可以在浏览播放空间时获得 "sonic 视差"。In the Pantheon scene, for example, the sound of the fountain was placed as a separate emitter close enough to the user that they could get a sense of "sonic parallax" as they walk around the play space. 在秘鲁的 Salinas de Maras 场景中,每个小流的声音被置为单独的发射器,以构建一个更具沉浸的环境环境,使用户能够使用该位置的真实声音。In Peru's Salinas de Maras scene, the sounds of individual little streams were placed as separate emitters to build a more immersive ambient environment, surrounding the user with the authentic sounds of that location.
  • 样条发射器: 这些发射器会根据它们所附加到的对象的视觉位置在3D 空间中移动。Spline emitter: These emitters move in 3D space based on the visual position of the object that they're attached to. 一个例子就是 Machu Picchu 的一种训练,其中使用样条发射器给出了不同的方向性和运动。An example is the train in Machu Picchu, where we used a spline emitter to give a distinct sense of directionality and movement.
  • 音乐和 SFX: HoloTour 的某些方面表示更风格或迷人的方法,它使用音乐和声音效果来提高激动人心的影响力。Music and SFX: Certain aspects of HoloTour that represent a more stylized or cinematic approach use music and sound effects to heighten emotional impact. 例如,罗马教程结尾处的 gladiator 工作使用特殊效果,如 whooshes 和 stingers,以增强出现在场景中的标签的效果。For example, the gladiator battle at the end of the Rome tour uses special effects like whooshes and stingers to strengthen the effect of labels that appear in the scenes.

另请参阅See also