混合现实中的共享体验Shared experiences in mixed reality

全息影像不需要仅对一个用户保密。Holograms don't need to stay private to just one user. 全息版应用可能会从一个 HoloLens、iOS 或 Android 设备共享 空间锚点 ,使用户能够在真实世界的同一位置跨多个设备渲染一张全息影像。Holographic apps may share spatial anchors from one HoloLens, iOS, or Android device to another, enabling users to render a hologram at the same place in the real world across multiple devices.

定义共享方案的六个问题Six questions to define shared scenarios

在开始设计共享体验之前,定义目标方案非常重要。Before you begin designing for shared experiences, it’s important to define the target scenarios. 这些方案有助于阐明要设计的内容,并建立一个共同术语,以帮助比较和对比你的体验中所需的功能。These scenarios help clarify what you’re designing and establish a common vocabulary to help compare and contrast features required in your experience. 了解核心问题和解决方案的不同途径是发现此新介质固有机会的关键所在。Understanding the core problem, and the different avenues for solutions, is key to uncovering opportunities inherent in this new medium.

通过我们的 HoloLens 合作伙伴机构的内部原型和探索,我们创建了六个问题来帮助你定义共享方案。Through internal prototypes and explorations from our HoloLens partner agencies, we created six questions to help you define shared scenarios. 这些问题形成了一个框架,而不是一种全面的设计,旨在帮助用户更好地提取方案的重要属性。These questions form a framework, not intended to be exhaustive, to help distill the important attributes of your scenarios.

1. 它们如何共享?1. How are they sharing?

演示可能由单个虚拟用户导致,而多个用户可进行协作,或者教师可以向使用虚拟材料的虚拟学生提供指导,这种体验的复杂程度取决于用户在方案中具有或可以拥有的代理级别。A presentation might be led by a single virtual user, while multiple users can collaborate, or a teacher might provide guidance to virtual students working with virtual materials—the complexity of the experiences increases based on the level of agency a user has or can have in a scenario.

表中的 holograph

有多种方法可以共享,但我们发现,其中的大多数方法分为三类:There are many ways to share, but we’ve found that most of them fall into three categories:

  • 演示:在向多个用户显示相同内容时。Presentation: When the same content is being shown to several users. 例如:教授使用同一个全息材料向每个人提供一个讲座给多名学生。For example: A professor is giving out a lecture to several students using the same holographic material being presented to everyone. 不过,教授可能会有自己的提示和注释,而其他人可能看不到这些内容。The professor, however, could have his/her own hints and notes that may not be visible to others.
  • 协作:当人们共同合作来实现某些共同的目标时。Collaboration: When people are working together to achieve some common goals. 例如:专家提供了一个项目,以了解如何进行心率。For example: The professor gave out a project to learn about doing a heart surgery. 学生配对并创建共享的技能实验室体验,使医疗学生可以在心形模型上进行协作并学习。Students pair up and create a shared skills lab experience, which allows medical students to collaborate on the heart model and learn.
  • 指南:当一个人帮助他人解决一种更一对一的样式交互中的问题时。Guidance: When one person is helping someone to solve a problem in a more one-to-one style interaction. 例如:当用户在共享体验中执行 "视觉" 操作技能实验室时,为学生提供指导。For example: The professor giving guidance to a student when he/she is doing the heart surgery skills lab in the shared experience.

2. 组大小是多少?2. What is the group size?

一对一 共享体验可提供强大的基线,理想情况下,可以在此级别创建概念证明。One-to-one sharing experiences can provide a strong baseline and ideally your proofs of concept can be created at this level. 但要注意的是, (超过六名用户的大型组共享) 可能会导致技术 (数据和网络) ,同时社交,同时还会遇到 多个头像 (所造成的影响。But be aware that sharing with large groups (beyond six people) can lead to difficulties both technical (data and networking) and social (the impact of being in a room with several avatars). 当你从 小****组到大型组 时,复杂性会以指数方式增加。Complexity increases exponentially as you go from small to large groups.

我们发现组的需求可能分为三个大小类别:We have found that the needs of groups can fall into three size categories:

  • 1:11:1
  • 小型 < 7Small < 7
  • 大型 >= 7Large >= 7

组大小对重要问题有影响,因为它会影响:Group size makes for an important question because it influences:

  • 全息空间中人员的表示形式Representations of people in holographic space
  • 对象的小数位数Scale of objects
  • 环境规模Scale of environment

3. 每个人是谁?3. Where is everyone?

当在同一位置发生共享体验时,混合现实的强度会成为一种情况。The strength of mixed reality comes into play when a shared experience can take place in the same location. 我们 称之为这一 定位。We call that colocated. 相反,当分配组时,至少有一个参与者不在同一物理空间中 (例如,在使用) VR 的情况下,我们称之为 远程体验Conversely, when the group is distributed and at least one participant isn't in the same physical space (as is often the case with VR) we call that a remote experience. 通常情况下,你的组具有协同工作的参与者和远程参与者 (例如,会议室) 的 两个组Often, it’s the case that your group has both colocated and remote participants (for example, two groups in conference rooms).

表上有 holograph 的三个用户

以下类别有助于传达用户所在的位置:Following categories help convey where users are located:

  • 定位 :所有 用户都将处于相同的物理空间。Colocated: All your users will be in the same physical space.
  • 远程:所有用户将位于不同的物理空间。Remote: All your users will be in separate physical spaces.
  • 两者:你的用户将混合使用协同定位和远程共享空间。Both: Your users will be a mix of colocated and remote spaces.

此问题很重要,因为它会影响:This question is crucial because it influences:

  • 用户如何通信?How people communicate?
    • 例如:是否应使用头像?For example: Whether they should have avatars?
  • 他们看到的对象。What objects they see. 是否共享所有对象?Are all objects shared?
  • 是否需要适应自己的环境?Whether we need to adapt to their environment?

4. 他们何时共享?4. When are they sharing?

我们通常会考虑一下共享体验时的 同步 体验:我们都在一起。We typically think of synchronous experiences when shared experiences come to mind: We’re all doing it together. 但如果包含由其他人添加的单个虚元素,则可以使用 异步 方案。But if we include a single, virtual element that was added by someone else, we have an asynchronous scenario. 在虚拟环境中,假设有一个便笺或语音备忘录。Imagine a note, or voice memo, left in a virtual environment. 如何在设计中处理100虚拟备忘录?How do you handle 100 virtual memos left on your design? 如果他们来自具有不同隐私级别的数十个用户怎么办?What if they’re from dozens of people with different levels of privacy?

假设你的体验是以下一类时间:Consider your experiences as one of these categories of time:

  • 同步:同时共享全息体验。Synchronously: Sharing the holographic experience at the same time. 例如:两名学生同时执行技能实验室。For example: Two students doing the skills lab at the same time.
  • 以异步方式:在不同时间共享全息体验。Asynchronously: Sharing the holographic experience at different times. 例如:两名学生执行技能实验室,但在不同时间使用单独的部分。For example: Two students doing the skills lab but working on separate sections at different times.
  • 两者:你的用户有时会以同步方式同步,但会以异步方式进行共享。Both: Your users will sometimes be sharing synchronously but other times asynchronously. 例如:一名教授在以后对学生完成分配的教授,并在下一天为学生留下笔记。For example: A professor grading the assignment done by the students at a later time and leaving notes for students for the next day.

此问题很重要,因为它会影响:This question is important because it influences:

  • 对象和环境暂留。Object and environment persistence. 例如:存储状态,以便可以检索它们。For example: Storing the states so they can be retrieved.
  • 用户透视。User perspective. 例如:在离开便笺时,可能会记住用户正在查看的内容。For example: Perhaps remembering what the user was looking at when leaving notes.

5. 他们的物理环境有多相似?5. How similar are their physical environments?

除非这些环境的设计是相同的,否则两个完全相同的现实环境(即在协同工作的体验之外)的可能性很小。The likelihood of two identical real-life environments, outside of colocated experiences, is slim unless those environments have been designed to be identical. 更有可能拥有 类似 的环境。You’re more likely to have similar environments. 例如,会议室是类似的,他们通常会有一个集中定位的表。For example, conference rooms are similar—they typically have a centrally located table surrounded by chairs. 另一方面,生活场所是不同的 * *,可以在无限布局数组中包含任意数量的家具。Living rooms, on the other hand, are dissimilar** and can include any number of pieces of furniture in an infinite array of layouts.

表上的 Holograph

假设你的共享体验符合以下两个类别之一:Consider your sharing experiences fitting into one of these two categories:

  • 类似:往往具有相似家具、环境光线和声音、物理房间大小的环境。Similar: Environments that tend to have similar furniture, ambient light and sound, physical room size. 例如:教授在讲座厅 A 中,学生处于讲座厅 B。讲座厅的椅子可能比 B 小,但这两家公司可能都有物理办公桌来放置全息影像。For example: Professor is in lecture hall A and students are in lecture hall B. Lecture hall A might have fewer chairs than B but they both may have a physical desk to place holograms on.
  • 不同:家具 设置、房间大小、光源和声音注意事项中的环境不同。Dissimilar: Environments that are different in furniture settings, room sizes, light, and sound considerations. 例如:教授在一个侧重点房间里,学生们处于大讲座厅,并与学生和老师一起填写。For example: A professor is in a focus room, but students are in a large lecture hall, filled with students and teachers.

考虑环境很重要,因为它会影响:It's important to think about the environment, as it will influence:

  • 用户将如何体验这些对象。How people will experience these objects. 例如:如果你的体验最适用于表,并且用户没有表?For example: If your experience works best on a table and the user has no table? 或在平面地面表面上,但用户的空间很混乱。Or on a flat floor surface but the user has a cluttered space.
  • 对象的小数位数。Scale of the objects. 例如:在表中放置一个六英尺的人类模型可能具有挑战性,但心形模型会很好。For example: Placing a six feet human model on a table could be challenging but a heart model would work great.

6. 它们使用哪些设备?6. What devices are they using?

今天,你通常可能会看到两个 沉浸式设备 之间的共享体验 (这些设备对于按钮和相对功能可能略有不同,但在这些设备上,如果解决方案面向这些设备,则不会极大地) 或两个 全息设备Today you’re often likely to see shared experiences between two immersive devices (those devices might differ slightly For buttons and relative capability, but not greatly) or two holographic devices given the solutions being targeted at these devices. 但请考虑是否需要将 2d 设备 (移动/桌面参与者或观察者) 将是必需的注意事项,尤其是在 混合2d 和3d 设备 的情况下。But consider if 2D devices (a mobile/desktop participant or observer) will be a necessary consideration, especially in situations of mixed 2D and 3D devices. 了解你的参与者将使用的设备类型很重要,不仅因为它们采用不同的保真度和数据约束和机会,而且是因为用户对每个平台都有独特的期望。Understanding the types of devices your participants will be using is important, not only because they come with different fidelity and data constraints and opportunities, but because users have unique expectations for each platform.

了解共享体验的潜力Exploring the potential of shared experiences

可以结合上述问题的答案来更好地了解你的共享方案,并在扩展体验时 crystallizing 问题。Answers to the questions above can be combined to better understand your shared scenario, crystallizing the challenges as you expand the experience. 对于 Microsoft 的团队而言,这有助于建立道路地图来改善我们目前使用的体验、了解这些复杂问题的细微差别,以及如何利用混合现实中的共享体验。For the team at Microsoft, this helped establish a road map for improving the experiences we use today, understanding the nuance of these complex problems and how to take advantage of shared experiences in mixed reality.

例如,请考虑从 HoloLens 启动的 Skype 方案之一:用户使用了如何使用远程定位的专家的帮助 修复损坏的光源交换机For example, consider one of Skype’s scenarios from the HoloLens launch: a user worked through how to fix a broken light switch with help from a remotely located expert.

通过 Skype for HoloLens 修复带有协助的光源开关

专家提供从其 2d 桌面计算机到 三维混合现实 设备用户的 1:1 指导。本 指南同步 的,物理环境是 不同 的。An expert provides 1:1 guidance from his 2D, desktop computer to a user of a 3D, mixed-reality device. The guidance is synchronous and the physical environments are dissimilar.

这种体验就是我们当前经验的一项变化,即,将视频和语音的模式应用到新媒体。An experience like this is a step-change from our current experience—applying the paradigm of video and voice to a new medium. 但正如我们所期待的,我们必须更好地定义应用场景和构建体验的机会,以反映混合现实的强度。But as we look to the future, we must better define the opportunity of our scenarios and build experiences that reflect the strength of mixed reality.

请考虑由 NASA 的 Jet 喷气实验室开发的 OnSight 协作工具Consider the OnSight collaboration tool, developed by NASA’s Jet Propulsion Laboratory. 使用 Mars 火星任务中数据的科学家可在 Martian 横向的数据内实时与同事进行协作。Scientists working on data from the Mars rover missions can collaborate with colleagues in real time within the data from the Martian landscape.

在以远程方式分开的同事之间进行协作,为 Mars 火星计划工作

科研人员 使用三维****和二维 设备,通过一组较 远程 同事来浏览环境。**协作* (同步,但可以异步方式进行) 并且物理环境 (虚拟) 类似。*A scientist explores an environment using a 3D, mixed-reality device with a small group of remote colleagues using 3D and 2D devices. The collaboration is synchronous (but can be revisited asynchronously) and the physical environments are (virtually) similar.

像 OnSight 这样的体验会带来新的合作机会。Experiences like OnSight present new opportunities to collaborate. 从物理上指出虚拟环境中的元素,并在其解释其观点时共享其外观。From physically pointing out elements in the virtual environment to standing next to a colleague and sharing their perspective as they explain their findings. OnSight 使用浸入式和状态的镜头来重新考虑混合现实中的共享体验。OnSight uses the lens of immersion and presence to rethink sharing experiences in mixed reality.

直观的协作是会话的成为,合作并了解我们如何将此直觉应用到混合现实的复杂性至关重要。Intuitive collaboration is the bedrock of conversation, working together and understanding how we can apply this intuition to the complexity of mixed reality is crucial. 如果我们不仅可以在混合现实中重新创建共享体验,还可以提升它们,但对于未来的工作来说,这是一项转变。If we can not only recreate sharing experiences in mixed reality but supercharge them, it will be a paradigm shift for the future of work. 在混合现实中设计共享体验是一项新的令人兴奋的空间-我们只是一开始。Designing for shared experiences in mixed reality is new and exciting space—and we’re only at the beginning.

开始构建共享体验Get started building shared experiences

根据您的应用程序和方案,将有各种要求来实现您所需的体验。Depending on your application and scenario, there will be various requirements to achieve your desired experience. 其中包括:Some of these include:

  • Match:创建会话、播发会话、发现和邀请特定人员,同时在本地和远程加入会话。Match-making: Ability to create sessions, advertise sessions, discover and invite specific people, both locally and remotely to join your session.
  • 定位点共享:能够在一个公共的本地空间内对齐多个设备上的坐标,因此对于所有用户,全息影像都显示在同一位置。Anchor sharing: Ability to align coordinates across multiple devices in a common local space, so holograms appear in the same place for all people.
  • 网络:能够在所有参与者之间实时同步用户和全息影像的位置、交互和移动。Networking: Ability to have positions, interactions, and movements of people and holograms synchronized in real time across all participants.
  • 状态存储:能够在空间中存储全息图的特性和位置,以便进行中型会话加入,稍后重新调用,并抵御网络问题。State storage: Ability to store hologram characteristics and locations in space for mid-session join, recall at a later time, and robustness against network issues.

共享体验的关键是,多个用户在自己的设备上看到同一个全息影像,这通常通过共享锚点来跨设备对齐坐标来完成。The key to shared experiences is having multiple users seeing the same holograms in the world on their own device, frequently done by sharing anchors to align coordinates across devices.

若要共享锚,请使用 Azure 空间锚点To share anchors, use the Azure Spatial Anchors:

  • 首先,用户放置全息影像。First the user places the hologram.
  • 应用创建 空间锚,以便在世界上准确固定该全息图。App creates a spatial anchor, to pin that hologram precisely in the world.
  • 可以通过 Azure 空间锚点将锚点共享到 HoloLens、IOS 和 Android 设备。The anchors can be shared to HoloLens, iOS, and Android devices via Azure Spatial Anchors.

使用共享空间定位点,每个设备上的应用现在都有一个可放置内容的 通用坐标系统With a shared spatial anchor, the app on each device now has a common coordinate system in which they can place content. 现在,应用可以确保在同一位置定位和定位全息影像。Now the app can ensure to position and orient the hologram at the same location.

在 HoloLens 设备上,你还可以从一个设备离线共享锚点。On HoloLens devices, you can also share anchors offline from one device to another. 使用以下链接来确定最适合你的应用程序。Use the links below to decide what's best for your application.

评估技术选项Evaluating tech options

有各种服务和技术选项可用于帮助构建多用户混合现实体验。There are various service and technology options available to help build multi-user mixed reality experiences. 选择路径可能比较困难,因此,要采用以方案为中心的角度,下面详细介绍了一些选项。It can be tricky to choose a path, so taking a scenario-focused perspective, some options are detailed below.

共享静态全息影像 (无交互) Shared static holograms (no interactions)

在应用中利用 Azure 空间锚Leverage Azure Spatial Anchors in your app. 通过跨设备启用并共享空间锚,你可以创建一个应用程序,让用户同时查看同一位置的全息影像。Enabling and sharing spatial anchors across devices allows you to create an application where users see holograms in the same place at the same time. 需要跨设备进行额外的同步,以使用户能够与全息影像交互,并查看影像的移动或状态更新。Additional syncing across devices is needed to enable users to interact with holograms and see movements or state updates of holograms.

共享第一人称视角Share first person perspective

如果有受支持的 Miracast 接收方(如 PC 或电视),则对本地用户利用内置 Miracast 支持,无需其他应用代码。Leverage built-in Miracast support, for local users when you have a supported Miracast receiver, such as a PC or TV – no additional app code is needed.

在应用中利用 MixedReality-WebRTC ,对于远程用户,或在你具有要共享的非 Miracast 设备时。Leverage MixedReality-WebRTC in your app, for remote users or when you have non-Miracast devices that you’d like to share to. 启用 WebRTC 连接将在用户之间启用1:1 音频/视频流,并使用数据通道跨设备进行消息传送。Enabling a WebRTC connection enables 1:1 audio/video streams between users, with a data channel for messaging across devices, as well. 混合现实实现通过向其他人提供 HoloLens 用户视图的混合现实视频流,针对 HoloLens 进行优化。The mixed reality implementation optimizes for HoloLens, by providing mixed reality capture video stream of the view of the HoloLens user to others. 如果要将视频流扩展到多个远程客户端,则通常使用一个 MCU 服务提供程序 (Multipoint 会议单位) ,如 SignalWireIf you desire to scale up video streaming to multiple remote clients, an MCU service provider (Multipoint Conferencing Unit) is typically used, such as SignalWire. 可通过 Freeswitch获取对 Azure 的一键式 SignalWire 部署。A one-click SignalWire deployment to Azure is available via Freeswitch.

备注

请注意,SignalWire 是付费服务,不属于 Microsoft。Please note that SignalWire is a paid service and is not owned/affiliated with Microsoft.

Presenter-Spectator 应用程序和演示Presenter-Spectator applications and Demos

利用 MixedReality-SpectatorViewspectator view 功能 引入应用。Leverage MixedReality-SpectatorView to bring spectator view functionality into your app. 启用 (HL、Android、iOS 和摄像机) 的其他设备,以查看 HoloLens 在同一位置的不同透视中看到的内容,并接收有关与全息影像交互的主机 HoloLens 用户交互的更新。Enable other devices (HL, Android, iOS, and video cameras) to see what the HoloLens sees from a different perspective in the same location, and receive updates on interactions of the host HoloLens user interacting with the holograms. 使用同一个应用的 spectator 伴随空间,观看、拍摄照片并记录主机对应用程序中的全息影像执行的操作的视频。Watch, take pictures, and record video of what the host does with the holograms in the application from your own spatial perspective with the spectator companion of the same app.

注意: 图片是通过 iOS/Android 设备上的屏幕截图拍摄的。Note: Pictures are taken via screenshot on iOS/Android devices.

多用户协作体验Multi-user collaborative experience

首先介绍多用户学习教程,该教程利用了用于本地用户和Photon SDKAzure 空间锚点来同步场景中的内容/状态。Start with our multi-user learning tutorial, which leverages Azure Spatial Anchors for local users and Photon SDK for syncing the content/state in the scene. 创建本地协作应用程序,其中每个用户在场景中的全息影像上都有自己的观点,每个用户都可以完全与全息影像交互。Create locally collaborative applications where each user has his/her own perspective on the holograms in the scene and can each fully interact with the holograms. 所有设备都提供更新,交互冲突管理由 Photon 处理。Updates are provided across all devices and interaction conflict management is handled by Photon.

备注

请注意, Photon 是一种非 Microsoft 产品,因此,可能需要使用 Photon 进行计费关系,以实现更高的使用量。Please note that Photon is a non-Microsoft product, so a billing relationship with Photon may be required to productize and scale for higher usage.

未来工作Future work

组件功能和接口将帮助提供跨各种方案和基础技术的常见一致性和可靠支持。Component capabilities and interfaces will help in providing common consistency and robust support across the various scenarios and underlying technologies. 在此之前,请选择与你要在应用程序中尝试实现的方案相符的最佳路径。Until then, choose the best path that aligns to the scenario you're trying to achieve in your application.

不同的方案或希望使用不同的技术/服务?Different scenario or desire to use a different tech/service? 请在此页底部的相应存储库中提供 GitHub 问题的反馈,或在 HoloDevelopers 的时差上联系。Provide feedback as GitHub issues in the corresponding repo, at the bottom of this page, or reach out on HoloDevelopers slack.

另请参阅See also