设计全息影像的过程The making of Designing Holograms

备注

请在此页上为所有的冷 Gif 和嵌入视频提供小的加载窗口。Please allow for a small loading window to account for all the cool GIFs and embedded videos on this page.

了解如何设计混合现实很难,因为介质并不一定能顺利地转换为2D 设计过程。Learning how to design for mixed reality can be hard because the medium doesn't always translate well to 2D design processes. 在 Microsoft,我们为 HoloLens 2 创建了免费应用,帮助你了解混合现实 UX 设计亲身的基础知识。Here at Microsoft, we've created a free app for the HoloLens 2 to help you learn the fundamentals of mixed reality UX Design firsthand. 设计全息影像应用程序的独特方法深入混合现实行为、提示和建议,帮助你创建自己的富有吸引力的、令人惊叹的 HoloLens 应用。The unique approach of the Designing Holograms app dives into mixed reality behaviors, tips, and recommendations to help you create engaging and amazing HoloLens apps of your own. 从 Microsoft Store 免费下载应用,并从 Microsoft 混合现实设计团队那里了解!Download the app for free from the Microsoft Store and learn from Microsoft’s Mixed Reality Design Team!


设计全息图的演示房间中标题跟踪场景的动画 GIF

设计全息图的演示房间 (也称为娃房子)Designing Hologram’s demo room (also known as the doll house)

混合现实设计Designing for mixed reality

与许多人一样,我使用的是设计移动应用。Like many of you, I used to design mobile apps. 从2D 设计世界来,跳转到空间计算的完整,其中一切现在都在世界中,这是一项重大转变。Coming from a 2D design world, jumping into full on spatial computing, where everything is now in the world, was a significant shift. 在混合现实中,应用不再局限于2D 屏幕;事实上,它们几乎是免费的,放在现实世界中,并与真实对象进行交互。In mixed reality, apps aren't confined to a 2D screen anymore; in fact, they're almost free, placed in the real world and interacting with real objects.

对于我来说,将3D 体验连接到传统的2D 设计过程是混合现实开发的最具挑战性的方面。To me, connecting 3D experiences to conventional 2D design processes is the most challenging aspect of mixed reality development. 在与客户的对话中,我知道 "我知道要包含哪些功能以及如何让它们启动并运行"。In conversations with customers, I would hear things like “I know what features to include and how to get them up and running. 就是代码,我可以按照文档和教程操作,而是在用户体验方面?It’s code, I can follow the docs and tutorials, but the user experience? 如此多的功能、不同的输入选项、不同的方案和物理环境,这一点很惊人。So many features, different input options, different scenarios, and physical environments, it’s overwhelming".

从圣马力诺2中的 HoloLens 2 设计研讨会开始的图像,位于旧金山 2 的 Hololens 2 设计研讨会Image from the HoloLens 2 Design Workshop in San Francisco Image from the HoloLens 2 Design Workshop in San Francisco

机会An opportunity to teach

这并不是很明显,但会出现一个极好的机会,将混合现实用作一种用于教授的媒介。It wasn’t obvious at first, but an excellent opportunity was presented to use mixed reality as a Medium to teach it.

设计全息影像是介绍混合现实设计概念和建议的视觉体验。Designing Holograms is a visual experience that explains mixed reality design concepts and recommendations. 这只是你以及演示混合现实设计概念的虚拟老师。It’s just you and a virtual teacher demonstrating mixed reality design concepts. 一切都是从第三人的角度来看,它在自己的空间中实现了丰富的体验。Everything is from a third person perspective with the experience firmly in your own space.


设计全息影像尾端视频Designing Holograms trailer video

探索娃房子Exploring the doll house

娃房子是我们在整个应用程序中使用的虚拟环境。The doll house is the virtual environment we use throughout the app. 环境为 80 x 60 x 40-cm,其中包含大多数房间共有的基本元素,如墙壁、灯、家具、桌子和电视。The environment is an 80 x 60 x 40-cm miniature room that contains the basic elements that most rooms have in common, like walls, lamps, furniture, a table, and a TV. 娃房子是应用程序体验的主要 protagonist,因此我们需要确保它在任何环境中都能正常工作。The doll house is the main protagonist of the app experience, so we needed to make sure it would work great in any environment. 将其视为一个小型演示空间,用于可视化各种混合现实概念。Think of it as a small demo room for visualizing all sorts of mixed reality concepts.

Dollhouse 调整行为的视频Video of the Dollhouse adjustment behavior

1:1 与1:10 原型1:1 vs 1:10 prototypes

我们最初假设,1:1 的演示会令人吃惊,几乎与查看真实的老师一样。Our initial assumption was that 1:1 demonstrations would be amazing, almost like looking at a real life teacher. 用户将看到老师在实际生活中看到的一切内容。The user would see everything that the teacher sees at a real life scale. 但是,我们立即意识到存在以下几个问题:However, we immediately realized that there would be a few problems:

  • 大多数开发人员会在办公室或比演示房间小的房间内运行应用程序,因此不适合。Most developers will run their apps in offices, or rooms smaller than the demo room, so it wouldn’t fit.
  • 显示是累加的,这意味着整个虚拟环境将覆盖在用户的空间上。Displays are additive, meaning the entire virtual environment will be overlaid over a user’s room. 这可能会使两个表(可能为 double couches)和不对齐的墙壁产生混淆。That can get confusing with two tables, maybe double couches, and walls that don’t align.
  • 并且最糟糕的是,所有虚拟环境都受视图的字段约束。And worst of all a virtual environment heavily constrained by a field of view.

当我们尝试使用微型1:10 刻度时,结果是一个极好的鸟瞰视图。When we tried out a mini 1:10 scale, the result was a fantastic birds-eye view of a realistic room. 你可以从任何角度查看所有内容。You could see everything that was going on from any angle all at the same time. 最令人吃惊的是,大多数测试人员发现它很多地都可以看到较小的版本,然后他们永远无法切换回1:1 规模。What was most surprising is that most testers found it so much more immersive to see a small version, then they never toggled back to the 1:1 scale. 我们决定实际会弃用1:1 版本,并避免在 UI 和应用程序的其他方面进行调整所需的额外工作。So we decided to actually scrap the 1:1 version and avoid the extra work required to adapt UI and other aspects of the app.

具有 1:1 刻度的 1:1 缩放字段视图的视图字段Field of view with 1:1 scale Field of view with 1:1 scale

具有 1:10 刻度的 1:10 缩放字段视图的视图字段Field of view with 1:10 scale Field of view with 1:10 scale

使用混合现实捕获Using Mixed Reality Capture

此应用程序的最重要特性之一是使用混合现实捕获来讲授和演示混合现实设计概念。One of the most characteristic features of this app is the use of Mixed Reality Capture to teach and demonstrate mixed reality design concepts.

Microsoft 的 San 中有混合现实的捕获工作室。Microsoft has a Mixed Reality Capture studio in San Francisco. Microsoft 还向其他工作室提供此技术,其中包括华盛顿特区的头像、洛杉矶、伦敦的 Dimension 工作室、首尔的 SK 电信和柏林上的 Volucap。Microsoft also licenses this technology to other studios, which include Avatar Dimension in Washington D.C., Metastage in Los Angeles, Dimension Studios in London, SK Telecom in Seoul, and Volucap in Berlin. 可在此处找到有关 混合现实捕获工作室的详细信息。You can find more information on our Mixed Reality Capture Studios here.


在 San 中,混合现实中的一106照相机的 Daniel Escudero 的原始胶片。Raw footage of Daniel Escudero from one of the 106 cameras in the Mixed Reality Capture Studio in San Francisco.

捕获过程将生成一个 keyframed 的网格、法线和纹理,可将其作为 "OBJ/PNG" 文件传送,以便进一步后期生产,或准备好将其作为 "The capture process generates a keyframed mesh, normals, and texture, which can be delivered as OBJ/PNG files for further post-production, or ready for playback as an H.264 compressed MP4 file. 这些文件可以导入到 Unity、Unreal、本机和 WebXR 项目中。These files can be imported into Unity, Unreal, Native, and WebXR projects. 文件可在 Windows、iOS、Mac、Android、幻 Leap 和 Playstation VR 上运行。Files can run on Windows, iOS, Mac, Android, Magic Leap, and Playstation VR.


提供的捕获播放机用于分析包含音频和嵌入网格的视频的 mp4。The Capture Player provided to analyze mp4s that contain video with audio and embedded meshes.

操作捕获和虚拟对象Manipulating captures and virtual objects

混合现实捕获会生成人员或动物的虚拟表示形式,但有时您可能需要这些字符与其他虚拟对象进行交互。Mixed Reality Captures produce virtual representations of people or animals, but at times you may need those characters to interact with other virtual objects. 下面两个示例演示了我们处理场景以实现此效果的不同方式。The following two examples show different ways we manipulated the scenes to achieve this effect.

打印头注视调整Head Gaze Adjustment

Headgaze 调整使你可以在运行时移动已捕获人员的标题,这意味着你可以获得用户的捕获面。Headgaze adjustment lets you move a captured person’s head at runtime, meaning you could have a capture face towards a user. 在我们的示例中,我们使用它来显示所需的视图和字段的字段。In our case, we used it to show the field of view and field of interest. 下面是一个移动 gameobject,它作为打印头注视的目标来查看。What you see below is a moving gameobject acting as a target for the head gaze to look at. 当我们将目标从一侧移到另一侧时,捕获的开头将如下所示。As we move the target from side to side, the head of the capture follows.

我们使用这一技巧来确保空闲捕获始终面对存放在娃房子不同部分的全息影像。We used this trick to make sure that the idle capture would always face towards holograms placed in different parts of the doll house.

在 Unity 中的目标 gameobject 之后,在运行时移动捕获的头

当 Unity 中的目标 gameobject 后,将在运行时移动捕获的头。The Capture’s head being moved at runtime following a target gameobject in Unity.

同步动画对象Syncing Animated Objects

第二个是动画处理要与捕获移动同步的对象。The second one, was animating objects to sync with a capture’s movement. 在应用程序的不同部分,每隔五帧导入一个特定捕获的顺序 Obj。In different parts of the app, we imported sequential OBJs of a specific capture every five frames. 然后,Obj 在场景中进行动画处理,以确保它们与捕获的相应帧匹配。The OBJs were then animated in the scene to make sure they would match the corresponding frame of the capture. 这是一种令人厌烦的动画处理和 keyframing 的过程,但结果很好。It’s a tedious process of animating and keyframing, but the result is great. 你现在可以看到混合现实捕获与非捕获对象的交互。You can now see a Mixed Reality Capture interacting with non-captured objects.

在混合现实捕获和 UI 面板之间同步动画

在混合现实捕获和 UI 面板之间同步动画Synced animation between a Mixed Reality Capture and UI panel

UI 创作过程UI creative process

开始使用 UI 设计时,我们想要展示全息影像必须提供的一些神奇的做法。When we started the UI design, we wanted to show some of the magic and possibility that holograms have to offer. 在三维世界中,简单地显示静态2D 窗口和文本框并不合适。Simply showing static 2D windows and text boxes doesn’t feel right in the 3D world. 很多情况下不会显示,因此,从现在开始,我们决定从该位置移走,充分利用全息的3D 空间。Many of the possibilities at hand just don't show up, so right from the beginning we decided to move away from that and make full use of holographic 3D space.

首先,我们开始向面板、图标和文本信息添加一些粗细。At first, we started with adding some thickness to the panels, icons, and text information. 而且,作为用户,我看到的是一个文本框。Still, as a user, what I see is a text box. 包含图像的文本框,但不存在。Text boxes with images, but we aren't there. 我们将使用混合现实工具包 (MRTK) 着色器进行进一步的介绍。We went further by making use of the Mixed Reality Toolkit (MRTK) shaders. MRTK 着色器变成了一个功能强大的工具,并使用其模具功能向面板添加了消极的深度。The MRTK shaders became a powerful tool, and we made use of its stencil features to add negative depth to the panels. 这意味着,图标现在显示在透明面板的后面,而不是在文本框前面添加元素。That means instead of adding elements in front of a text box, the icons now appear behind a transparent panel. 我现在看到的是,我只是在现实世界中不能再复制,而这是一种开始使用全息幻的地方。What I see now as a user is something that I just can’t replicate anymore in the real world, and this is where holographic magic started to happen. 我也不是我想要阅读的,我还在现实世界中做了很多工作。Also as a user I don’t really like to read, I do a lot already in the physical world.

显然,图标的工作方式比简单文本更好,若要提供更为强大的指导,然后开始创建一组动画对象和头像,其中每个对象都告诉您在各自的方案中所执行的操作及其使用方式的一小部分。Obviously icons work a lot better than simple text does, to provide an even more powerful guidance, I then started creating a set of animated objects and avatars, each of them telling a tiny story about what is being done in the respective scenario and how it’s being used.

交互式全息菜单系统的动画 GIF

核心概念Core concepts

全息帧Holographic frame

用户的动画 GIF,其中突出显示了全息框架的 dollhouse

坐标系统Coordinate systems

用户使用突出显示的坐标系统围绕 dollhouse 的动画 GIF

眼动跟踪Eye tracking

用户的动画 GIF,其中突出显示了眼睛为眼睛的静止全息影像

房间扫描可视化和空间映射Room scan visualization and spatial mapping

要映射的 dollhouse 中的所有表面的动画 GIF

场景理解Scene understanding

正在识别的 dollhouse 中的对象的动画 GIF

手动射线的点和提交Point and commit with hand rays

用户的动画 GIF,其中突出显示了手 ray

"试用""Try it out" moments

设计全息影像会讲授混合现实概念,但它也允许您在房间内试用它们。Designing Holograms teaches mixed reality concepts, but it also allows you to try them in your room. 在其中一些说明后,我们将会暂停,并将您娃房子并进入互动时间。After some of those explanations, we pause and take you out of the doll house and into an interactive moment. 下面是这些交互式对话的一些示例:Here are some examples of those interactive moments:

手写帧的动画 GIF,其中显示了是否检测到指针以及何时进入视图的字段

在检测到指针和进入视图字段时显示的手写跟踪帧。The hand tracking frame showing when hands are detected and when they enter the field of view.

与碰撞 crystals 交互的动画 GIF

与 crystals 的交互发生交互Interacting with colliding crystals through far interaction

浏览近乎交互实用的动画 GIF

探索近交互实用Exploring near interaction affordances

关于团队About the team

Picture of Daniel Escudero Daniel EscuderoDaniel Escudero
领先技术设计人员Lead Technical Designer
Dan 是设计全息影像的创造性主管,当前作为旧金山的 Microsoft 混合现实院校的设计主管,以前是伦敦的 Microsoft 混合现实工作室之一中的设计器。Dan is the Creative Director on Designing Holograms and currently works as Design Lead for the Microsoft’s Mixed Reality Academy in San Francisco, and was previously a Designer in one of Microsoft’s Mixed Reality Studios in London.
Picture of Martin Wettig 圣马丁 WettigMartin Wettig
高级三维艺术家Senior 3D Artist
圣马丁使三维艺术和 UI 设计在设计全息影像上,以前是 Microsoft 的一个混合现实工作室的高级三维音乐家。Martin leads 3D Art and UI Design on Designing Holograms and was previously a Senior 3D Artist at one of Microsoft’s Mixed Reality Studios in Berlin.

非常感谢你对混合现实设计团队共享如此多的知识,并为 对象理论 的令人惊叹的用户提供有关项目中每个步骤的重要团队。A huge thank you to the Mixed Reality Design Team for sharing so much knowledge, and to the amazing folks at Object Theory for being essential teammates through every step of the project. 非常感谢您的热情,为您的热情和杰出的设计设计。Thank you all for you amazing talent, for your passion and exceptional eye for design.