空间映射Spatial mapping

空间映射提供了针对 HoloLens 环境中实际表面表面的详细表示,使开发人员能够创建一个具有说服力混合的现实体验。Spatial mapping provides a detailed representation of real-world surfaces in the environment around the HoloLens, allowing developers to create a convincing mixed reality experience. 通过将现实世界与虚拟世界合并,应用程序可以使全息影像看起来非常真实。By merging the real world with the virtual world, an application can make holograms seem real. 应用程序通过提供熟悉的实际行为和交互,还可以更自然地与用户的预期保持一致。Applications can also more naturally align with user expectations by providing familiar real-world behaviors and interactions.


设备支持Device supports

功能Feature HoloLens(第 1 代)HoloLens (1st gen) HoloLens 2HoloLens 2 沉浸式头戴显示设备Immersive headsets
空间映射Spatial mapping ✔️✔️ ✔️✔️

为什么空间映射很重要?Why is spatial mapping important?

利用空间映射,可以将对象放置在真实的图面上。Spatial mapping makes it possible to place objects on real surfaces. 这有助于在用户世界定位对象,并利用真实的深度提示。基于其他全息影像和现实世界对象 Occluding 全息影像有助于说服用户这些全息影像实际位于其空间中。This helps anchor objects in the user's world and takes advantage of real world depth cues. Occluding your holograms based on other holograms and real world objects helps convince the user that these holograms are actually in their space. 在空间中浮动或随用户移动的全息影像并不是真的。Holograms floating in space or moving with the user won't feel as real. 如果可能,可轻松放置项目。When possible, place items for comfort.

放置或移动全息影像时显示表面 (使用投影网格) 。Visualize surfaces when placing or moving holograms (use a projected grid). 这可以帮助用户了解他们最能最好地放置全息影像,并显示他们尝试放置全息影像的点是否未映射。This helps users know where they can best place their holograms, and shows if the spot they're trying to place the hologram isn't mapped. 如果用户的角度太多,您可以向用户 "布告栏项目"。You can "billboard items" toward the user if they end up at too much of an angle.

概念概述Conceptual overview

覆盖房间的网格面Mesh surfaces covering a room
覆盖房间的空间映射网格示例An example of a spatial mapping mesh covering a room

用于空间映射的两个主要对象类型为 "空间图观察器" 和 "空间图面"。The two primary object types used for spatial mapping are the 'Spatial Surface Observer' and the 'Spatial Surface'.

应用程序为空间图面观察器提供一个或多个边界卷,以定义应用程序希望接收空间映射数据的空间区域。The application provides the Spatial Surface Observer with one or more bounding volumes, to define the regions of space in which the application wishes to receive spatial mapping data. 对于每个卷,空间映射将为应用程序提供一组空间图面。For each of these volumes, spatial mapping will provide the application with a set of Spatial Surfaces.

这些卷可能在基于实际) 的固定位置 (固定位置,也可能附加到 HoloLens (,而不能在通过环境) 移动时使用 HoloLens 进行旋转。These volumes may be stationary (in a fixed location based on the real world) or they may be attached to the HoloLens (they move, but don't rotate, with the HoloLens as it moves through the environment). 每个空间图面都描述了小空间中的实际表面,表示为附加到世界锁定的 空间坐标系统的三角形网格。Each spatial surface describes real-world surfaces in a small volume of space, represented as a triangle mesh attached to a world-locked spatial coordinate system.

当 HoloLens 收集有关环境的新数据时,如果对环境进行了更改,空间图面将显示、消失,并发生变化。As the HoloLens gathers new data about the environment, and as changes to the environment occur, spatial surfaces will appear, disappear, and change.

空间映射与场景理解 WorldMeshSpatial Mapping vs. Scene Understanding WorldMesh

对于 HoloLens 2,可以使用 场景理解 SDK (EnableWorldMesh 设置) 来查询空间映射数据的静态版本。For HoloLens 2, it's possible to query a static version of the spatial mapping data using Scene understanding SDK (EnableWorldMesh setting). 下面是访问空间映射数据的两种方法之间的差异:Here are the differences between two ways of accessing the spatial mapping data:

  • 空间映射 API:Spatial Mapping API:
    • 有限范围:用户可使用大小受限的应用程序的空间映射数据。Limited range: the spatial mapping data available to applications in a limited size cached 'bubble' around the user.
    • 通过 SurfacesChanged 事件提供更改的网格区域的低延迟更新。Provides low latency updates of changed mesh regions through SurfacesChanged events.
    • 每个立方计量参数的三角形控制的详细信息级别。Variable level of details controlled by Triangles Per Cubic Meter parameter.
  • 场景理解 SDK:Scene understanding SDK:
    • 无限制范围-提供查询半径内所有扫描的空间映射数据。Unlimited range - provides all the scanned spatial mapping data within the query radius.
    • 提供空间映射数据的静态快照。Provides a static snapshot of the spatial mapping data. 获取更新的空间映射数据需要对整个网格运行新的查询。Getting the updated spatial mapping data requires running a new query for the whole mesh.
    • RequestedMeshLevelOfDetail 设置控制的详细级别的详细信息。Consistent level of details controlled by RequestedMeshLevelOfDetail setting.

哪些因素会影响空间映射质量?What influences spatial mapping quality?

此处详细介绍的几个因素可能会影响这些错误的频率和严重性。Several factors, detailed here, can affect the frequency and severity of these errors. 但是,你应设计应用程序,以便即使在空间映射数据中存在错误时,用户也可以实现其目标。However, you should design your application so that the user can achieve their goals even in the presence of errors in the spatial mapping data.

常见使用方案Common usage scenarios

常见空间映射使用方案的插图:放置、封闭、物理学和导航

放置Placement

空间映射为应用程序提供了向用户提供自然和熟悉的交互形式的机会。比将您的手机放在办公桌上的情况如何?Spatial mapping provides applications with the opportunity to present natural and familiar forms of interaction to the user; what could be more natural than placing down your phone on the desk?

(或更常见地限制全息影像的位置,) 的任何空间位置选择都是在表面上,从空间) 的 3D (点到图面 (上的 2D) 点的自然映射。Constraining the placement of holograms (or more generally, any selection of spatial locations) to lie on surfaces provides a natural mapping from 3D (point in space) to 2D (point on surface). 这会减少用户提供给应用程序所需的信息量,并使用户的交互速度更快、更简单且更精确。This reduces the amount of information the user needs to provide to the application and makes the user's interactions faster, easier, and more precise. 这是正确的,因为 "距离远离" 不是我们用于与其他人或计算机进行物理通信的东西。This is true because 'distance away' isn't something that we're used to physically communicating to other people or to computers. 当我们使用手指点时,我们指定的是一个方向,而不是距离。When we point with our finger, we're specifying a direction but not a distance.

下面是一个重要的注意事项:当应用程序从方向推断距离时 (例如,通过沿用户的注视方向 raycast 来查找最接近的空间图面) ,这必须生成用户可以可靠地预知的结果。An important caveat here's that when an application infers distance from direction (for example by doing a raycast along the user's gaze direction to find the nearest spatial surface), this must yield results that the user can reliably predict. 否则,用户将失去控制,这很快就会变得沮丧。Otherwise, the user will lose their sense of control and this can quickly become frustrating. 有助于实现此目的的一种方法是执行多个 raycasts,而不只是一个。One method that helps with this is to do multiple raycasts instead of just one. 聚合结果应更平滑且更具可预测性,不会受到暂时性的 "离群" 结果的影响 (因为这可能是由于光线通过小孔而导致的,或者是用户不知道) 的小位几何导致的。The aggregate results should be smoother and more predictable, less susceptible to influence from transient 'outlier' results (as can be caused by rays passing through tiny holes or hitting small bits of geometry that the user isn't aware of). 还可以在一段时间内执行聚合或平滑处理;例如,可以限制与用户之间的间距变化的最大速度。Aggregation or smoothing can also be performed over time; for example, you can limit the maximum speed at which a hologram can vary in distance from the user. 只是限制最小值和最大距离值也很有帮助,因此,要移动的全息影像不会突然飞入距离,也不会发生故障。Simply limiting the minimum and maximum distance value can also help, so the hologram being moved doesn't suddenly fly away into the distance or come crashing back into the user's face.

应用程序还可以使用图面的形状和方向来指导全息图放置。Applications can also use the shape and direction of surfaces to guide hologram placement. 全息椅子不应穿透墙壁,并应与地面齐平,即使它略有不同。A holographic chair shouldn't penetrate through walls and should sit flush with the floor even if it's slightly uneven. 这种功能可能会依赖于使用物理学冲突而不是 raycasts,但会应用类似的问题。This kind of functionality would likely rely upon the use of physics collisions rather than raycasts, however similar concerns will apply. 如果放置的全息图包含许多小多边形,如椅子上的支线,则将这些多边形的物理表示形式扩大到更宽和更平滑的位置,使其更能在无适合的情况下滑过空间图面。If the hologram being placed has many small polygons that stick out, like the legs on a chair, it may make sense to expand the physics representation of those polygons to something wider and smoother so that they're more able to slide over spatial surfaces without snagging.

在极大的情况下,可以完全简化用户输入,并可以使用空间表面来完成自动全息影像的放置。At its extreme, user input can be simplified away entirely and spatial surfaces can be used to do entirely automatic hologram placement. 例如,应用程序可能会在墙壁的某个位置放置一个全息片开关,以便用户按下。For example, the application could place a holographic light-switch somewhere on the wall for the user to press. 与可预测性相同的注意事项也适用于:如果用户期望控制全息图放置,但是应用程序不会始终在其预期位置放置全息影像 (如果光交换机出现在用户无法接触) 的某个位置,则这将是一项令人沮丧的体验。The same caveat about predictability applies doubly here; if the user expects control over hologram placement, but the application doesn't always place holograms where they expect (if the light-switch appears somewhere that the user can't reach), then this will be a frustrating experience. 它实际上可能更糟的是,自动放置需要用户更正一些时间,而不只是要求用户始终自行定位;由于 预期 的自动放置成功,手动更正就像是一种负担!It can actually be worse to do automatic placement that requires user correction some of the time, than to just require the user to always do placement themselves; because successful automatic placement is expected, manual correction feels like a burden!

另请注意,应用程序使用空间表面进行放置的能力很大程度上取决于应用程序的 扫描体验Note also that the ability of an application to use spatial surfaces for placement depends heavily on the application's scanning experience. 如果尚未扫描表面,则不能将其用于放置。If a surface hasn't been scanned, then it cannot be used for placement. 这是由应用程序为用户提供明确的选择,以便他们可以帮助扫描新的表面或选择新的位置。It's up to the application to make this clear to the user, so that they can either help scan new surfaces or select a new location.

在放置过程中,对用户的视觉反馈非常重要。Visual feedback to the user is of paramount importance during placement. 用户需要知道全息图基于具有 接地效果的最近图面。The user needs to know where the hologram is based on the nearest surface with grounding effects. 它们应了解为什么会对全息图的移动进行约束 (例如,因为与附近的另一个曲面) 发生冲突。They should understand why the movement of their hologram is being constrained (for example, because of collisions with another nearby surface). 如果他们无法在当前位置放置全息影像,则视觉对象反馈应清楚地说明为什么不这样做。If they can't place a hologram in the current location, then visual feedback should make it clear why not. 例如,如果用户尝试将全息的沙发卡在墙壁上,则墙后的沙发部分应 pulsate 为生气的颜色,。For example, if the user is trying to place a holographic couch stuck half-way into the wall, then the portions of the couch that are behind the wall should pulsate in an angry color. 相反,如果应用程序在用户可以看到真实表面的位置找不到空间图面,则应用程序应使其清晰。Or conversely, if the application can't find a spatial surface in a location where the user can see a real-world surface, then the application should make this clear. 此区域中明显不会产生接地效果的情况。The obvious absence of a grounding effect in this area may achieve this purpose.

封闭Occlusion

空间映射图面的主要用途之一就是遮蔽全息影像。One of the primary uses of spatial mapping surfaces is simply to occlude holograms. 这一简单的行为对全息影像的明显影响非常大,有助于创建 visceral 的意义,真正与用户 inhabits 相同的物理空间。This simple behavior has a huge impact on the perceived realism of holograms, helping to create a visceral sense that really inhabits the same physical space as the user.

封闭还向用户提供信息;当现实世界图面上出现全息图时,这就为世界上该全息图的空间位置提供了额外的视觉反馈。Occlusion also provides information to the user; when a hologram appears to be occluded by a real-world surface, this provides extra visual feedback as to the spatial location of that hologram in the world. 相反,封闭也可以有用 隐藏 用户的信息;occluding 的后台影像可以以直观的方式减少视觉干扰。Conversely, occlusion can also usefully hide information from the user; occluding holograms behind walls can reduce visual clutter in an intuitive way. 若要隐藏或显示全息图,用户只需移动其头。To hide or reveal a hologram, the user merely has to move their head.

封闭还可用于根据熟悉的物理交互,为自然用户界面提供合理的预期。如果某个图面的封闭像素是一个全息图,就是因为该表面是稳定的,因此用户应该认为,全息图将与该表面发生 冲突 ,而不会通过它。Occlusion can also be used to prime expectations for a natural user interface based upon familiar physical interactions; if a hologram is occluded by a surface it is because that surface is solid, so the user should expect that the hologram will collide with that surface and not pass through it.

有时,不需要封闭的全息影像。Sometimes, occlusion of holograms is undesirable. 如果用户需要与全息图交互,则需要对其进行查看,即使它位于实际的表面。If a user needs to interact with a hologram, then they need to see it - even if it is behind a real-world surface. 在这种情况下,在封闭像素 (例如,通过降低亮度) ,可以以不同的方式呈现此类全息图。In such cases, it usually makes sense to render such a hologram differently when it's occluded (for example, by reducing its brightness). 这样一来,用户就可以直观地找到全息影像,但他们仍然知道它位于某个地方。This way, the user can visually locate the hologram, but they'll still know it's behind something.

物理Physics

使用物理学模拟的另一种方法是使用空间映射来强化用户物理空间中 是否存在 全息影像。The use of physics simulation is another way in which spatial mapping can be used to reinforce the presence of holograms in the user's physical space. 当我的全息橡胶球从真实上掷走时,弹跳在地面上并消失,则很难相信这一点。When my holographic rubber ball rolls realistically off my desk, bounces across the floor and disappears under the couch, it might be hard for me to believe that it's not there.

物理学模拟还为应用程序提供了使用自然和熟悉的基于物理学的交互的机会。Physics simulation also provides the opportunity for an application to use natural and familiar physics-based interactions. 如果家具的响应方式与在楼层间滑动,并具有适当的惯性和摩擦,则用户可能会更容易地在地面上移动一张全息的家具。Moving a piece of holographic furniture around on the floor will likely be easier for the user if the furniture responds as if it were sliding across the floor with the appropriate inertia and friction.

若要生成真实的物理行为,你可能需要执行一些 网格处理 ,如填充孔、删除浮动 hallucinations 和平滑表面。To generate realistic physical behaviors, you'll likely need to do some mesh processing such as filling holes, removing floating hallucinations and smoothing rough surfaces.

还需要考虑应用程序的 扫描体验 如何影响其物理模拟。You'll also need to consider how your application's scanning experience influences its physics simulation. 首先,缺少的图面不会与任何内容发生冲突;当橡胶球向下 corridor 并从已知世界结束时,会发生什么情况呢?Firstly, missing surfaces won't collide with anything; what happens when the rubber ball rolls off down the corridor and off the end of the known world? 其次,您需要决定是否要在一段时间内继续对环境中的更改做出响应。Secondly, you need to decide whether you'll continue to respond to changes in the environment over time. 在某些情况下,你需要尽快做出响应;例如,如果用户使用门和家具作为 barricades 中的可移动的 tempest,则不会出现传入罗马字箭头的。In some cases, you'll want to respond as quickly as possible; say if the user is using doors and furniture as movable barricades in defense against a tempest of incoming Roman arrows. 但在其他情况下,你可能想要忽略新的更新;如果狗决定坐在轨迹的中间,则围绕地面的 racetrack 推动您的全息运动汽车就会突然不那么有趣。In other cases though, you may want to ignore new updates; driving your holographic sports car around the racetrack on your floor may suddenly not be so fun if your dog decides to sit in the middle of the track.

应用程序可以使用空间映射数据 (或代理授予全息字符,) 能够以真实人员的相同方式浏览现实。Applications can use spatial mapping data to grant holographic characters (or agents) the ability to navigate the real world in the same way a real person would. 这可以通过将其限制为用户及其好友的一组自然、熟悉的行为来帮助强化是否存在全息字符。This can help reinforce the presence of holographic characters by restricting them to the same set of natural, familiar behaviors as those of the user and their friends.

导航功能对于用户也很有用。Navigation capabilities could be useful to users as well. 在给定区域中生成导航地图后,可以将其共享,为不熟悉该位置的新用户提供全息说明。Once a navigation map has been built in a given area, it could be shared to provide holographic directions for new users unfamiliar with that location. 此映射可用于帮助保持行人的流量顺畅地流动,或避免在危险位置(如构造站点)中发生意外。This map could be designed to help keep pedestrian 'traffic' flowing smoothly, or to avoid accidents in dangerous locations like construction sites.

实现导航功能所涉及的关键技术挑战将会对不可的表面进行可靠检测, (人们不会对表进行检查! ) 并正常地适应环境中的变化 (人类不会遍历关闭的门! ) 。The key technical challenges involved in implementing navigation functionality will be reliable detection of walkable surfaces (humans don't walk on tables!) and graceful adaptation to changes in the environment (humans don't walk through closed doors!). 网格可能需要进行一些 处理 才能用于通过虚拟字符进行路径规划和导航。The mesh may require some processing before it's usable for path-planning and navigation by a virtual character. 平滑网格和删除 hallucinations 可帮助避免字符停滞。Smoothing the mesh and removing hallucinations may help avoid characters becoming stuck. 你可能还希望大幅简化网格,提高字符的路径规划和导航计算速度。You may also wish to drastically simplify the mesh to speed up your character's path-planning and navigation calculations. 在开发视频游戏技术时,这些挑战已经得到了很大的关注,其中提供了大量有关这些主题的信息。These challenges have received a great deal of attention in the development of video game technology, and there's a wealth of available research literature on these topics.

Unity 中的内置 NavMesh 功能不能与空间映射图面一起使用。The built-in NavMesh functionality in Unity cannot be used with spatial mapping surfaces. 这是因为在应用程序启动之前空间映射图面是未知的,但需要提前从源资产生成 NavMesh 数据文件。This is because spatial mapping surfaces aren't known until the application starts, but NavMesh data files need to be generated from source assets ahead of time. 另请注意,空间映射系统不会提供与用户当前位置 有关的有关表面的信息Also note that, the spatial mapping system won't provide information about surfaces far away from the user's current location. 因此,如果要构建大区域的地图,应用程序必须 "记住" 自身。So the application must 'remember' surfaces itself if it's to build a map of a large area.

可视化效果Visualization

大多数情况都适用于空间图面不可见;为了最大程度地减少视觉干扰,并让现实世界亲自讲述。Most of the time it's appropriate for spatial surfaces to be invisible; to minimize visual clutter and let the real world speak for itself. 但是,有时可以直接直观显示空间地图面,尽管它们的实际对等项是可见的。However, sometimes it's useful to visualize spatial mapping surfaces directly, despite their real-world counterparts being visible.

例如,当用户尝试在墙壁上放置全息式柜子时, (会将一只全息文件柜放到图面上时,假设) 通过将阴影转换到图面上来 "接地"。For example, when the user is trying to place a hologram onto a surface (placing a holographic cabinet on the wall, say) it can be useful to 'ground' the hologram by casting a shadow onto the surface. 这样,用户就可以更清楚地了解全息图和表面之间的确切物理距离。This gives the user a much clearer sense of the exact physical proximity between the hologram and the surface. 这也是在用户提交到用户之前,直观地预览更改的更通用的示例。This is also an example of the more general practice of visually 'previewing' a change before the user commits to it.

通过可视化图面,应用程序可以与用户共享其环境的了解。By visualizing surfaces, the application can share with the user its understanding of the environment. 例如,全息板游戏可能会将其标识为 "表" 的水平表面可视化,因此用户知道应该在何处交互。For example, a holographic board game could visualize the horizontal surfaces that it has identified as 'tables', so the user knows where they should go to interact.

可视化图面可用于向用户显示在视图中隐藏的附近的用户。Visualizing surfaces can be a useful way to show the user nearby spaces that are hidden from view. 这可能会提供一种方法,使用户能够从其客厅中访问其厨房 (及其包含的所有全息影像) 。This could provide a way to give the user access to their kitchen (and all of its contained holograms) from their living room.

空间映射提供的 surface 网格可能不是特别 "干净的"。The surface meshes provided by spatial mapping may not be particularly 'clean'. 必须对其进行适当的可视化。It's important to visualize them appropriately. 传统的照明计算可能会以视觉上分散的方式突出显示曲面法线中的错误,而投影到表面上的 "清除" 纹理可能有助于使其成为整齐的外观。Traditional lighting calculations may highlight errors in surface normals in a visually distracting manner, while 'clean' textures projected onto the surface may help to give it a tidier appearance. 在呈现表面之前,还可以执行 网格处理 来改善网格属性。It's also possible to do mesh processing to improve mesh properties, before the surfaces are rendered.

备注

HoloLens 2 实现了一种新的 场景了解运行时,它为混合现实开发人员提供了一个结构化、高级别的环境表示形式,旨在简化放置、封闭、物理学和导航的实现。HoloLens 2 implements a new Scene Understanding Runtime, that provides Mixed Reality developers with a structured, high-level environment representation designed to simplify the implementation of placement, occlusion, physics and navigation.

使用 Surface 观察器Using The Surface Observer

空间映射的起始点为 surface 观察程序。The starting point for spatial mapping is the surface observer. 程序流如下所示:Program flow is as follows:

  • 创建 surface 观察器对象Create a surface observer object
    • 提供一个或多个空间卷,以定义应用程序希望接收空间映射数据的感兴趣的区域。Provide one or more spatial volumes, to define the regions of interest in which the application wishes to receive spatial mapping data. 空间量只是定义空间区域的形状,例如球或 box。A spatial volume is simply a shape defining a region of space, such as a sphere or a box.
    • 使用带有世界锁定的空间坐标系统的空间量来识别物理世界的固定区域。Use a spatial volume with a world-locked spatial coordinate system to identify a fixed region of the physical world.
    • 使用空间量,使用正文锁定的空间坐标系统更新每个帧,以标识移动 (但不会向用户旋转) 的空间区域。Use a spatial volume, updated each frame with a body-locked spatial coordinate system, to identify a region of space that moves (but doesn't rotate) with the user.
    • 以后可以随时更改这些空间卷,因为应用程序或用户的状态会发生更改。These spatial volumes may be changed later at any time, as the status of the application or the user changes.
  • 使用轮询或通知来检索有关空间图面的信息Use polling or notification to retrieve information about spatial surfaces
    • 您可以随时 "轮询" 面观察器来实现空间图面状态。You may 'poll' the surface observer for spatial surface status at any time. 相反,你可以注册 surface 观察器的 "表面变化" 事件,这将在空间图面改变时通知应用程序。Instead, you may register for the surface observer's 'surfaces changed' event, which will notify the application when spatial surfaces have changed.
    • 对于动态空间量(如视图截锥或正文锁定卷),应用程序需要通过设置感兴趣的区域,然后获取当前的空间图面集来轮询每个帧的更改。For a dynamic spatial volume, such as the view frustum, or a body-locked volume, applications will need to poll for changes each frame by setting the region of interest and then obtaining the current set of spatial surfaces.
    • 对于静态卷(如涵盖单个房间的全球锁定的多维数据集),应用程序可能会注册 "已更改" 事件,以便在该卷内的空间表面发生变化时得到通知。For a static volume, such as a world-locked cube covering a single room, applications may register for the 'surfaces changed' event to be notified when spatial surfaces inside that volume may have changed.
  • 处理表面更改Process surfaces changes
    • 循环访问提供的空间图面集。Iterate the provided set of spatial surfaces.
    • 按添加、更改或删除的方式对空间图面进行分类。Classify spatial surfaces as added, changed, or removed.
    • 对于每个已添加或已更改的空间图面,如果适当,请提交一个异步请求,以在所需的详细信息级别接收表示该图面的当前状态的已更新网格。For each added or changed spatial surface, if appropriate submit an asynchronous request to receive updated mesh representing the surface's current state at the desired level of detail.
  • ) (以下部分中的详细信息,处理异步网格请求。Process the asynchronous mesh request (more details in following sections).

网格缓存Mesh Caching

空间图面由密集三角形网格表示。Spatial surfaces are represented by dense triangle meshes. 存储、呈现和处理这些网格可能会占用大量计算和存储资源。Storing, rendering, and processing these meshes can consume significant computational and storage resources. 因此,每个应用程序应采用适合于其需要的网格缓存方案,以最大程度地减少用于网格处理和存储的资源。As such, each application should adopt a mesh caching scheme appropriate to its needs, to minimize the resources used for mesh processing and storage. 此方案应该确定要保留哪些网格以及要丢弃哪些网格,以及何时更新每个空间图面的网格。This scheme should determine which meshes to keep and which to discard, and when to update the mesh for each spatial surface.

其中所述的许多注意事项将直接通知您的应用程序应该如何处理网格缓存。Many of the considerations discussed there will directly inform how your application should approach mesh caching. 当观察到不同的表面以及应捕获环境中的更改时,您应考虑用户在环境中的移动方式、需要的表面。You should consider how the user moves through the environment, which surfaces are needed, when different surfaces will be observed and when changes in the environment should be captured.

当解释 surface 观察器提供的 "表面变化" 事件时,基本网格缓存逻辑如下所示:When interpreting the 'surfaces changed' event provided by the surface observer, the basic mesh caching logic is as follows:

  • 如果应用程序看到之前未见过的空间表面 ID,则它应将其视为新的空间图面。If the application sees a spatial surface ID that it hasn't seen before, it should treat this as a new spatial surface.
  • 如果应用程序看到具有已知 ID 但具有新更新时间的空间图面,则它应将其视为更新的空间图面。If the application sees a spatial surface with a known ID but with a new update time, it should treat this as an updated spatial surface.
  • 如果应用程序不再看到具有已知 ID 的空间图面,则它应将其视为已删除的空间图面。If the application no longer sees a spatial surface with a known ID, it should treat this as a removed spatial surface.

然后,每个应用程序都可以进行以下选择:It's up to each application to then make the following choices:

  • 对于新空间图面,是否应请求网格?For new spatial surfaces, should mesh be requested?
    • 通常应立即为新的空间图面请求网格,这可能会向用户提供有用的新信息。Generally mesh should be requested immediately for new spatial surfaces, which may provide useful new information to the user.
    • 但是,在用户面前附近和之前的新空间图面应优先,并首先请求其网格。However, new spatial surfaces near and in front of the user should be given priority and their mesh should be requested first.
    • 如果不需要新网格,则为; 例如,如果应用程序永久或临时 "冻结" 其环境模型,则不应请求它。If the new mesh isn't needed, if for example the application has permanently or temporarily 'frozen' its model of the environment, then it shouldn't be requested.
  • 对于已更新的空间图面,是否应请求网格?For updated spatial surfaces, should mesh be requested?
    • 应将已更新的空间图面置于用户面前,并应首先请求其网格。Updated spatial surfaces near and in front of the user should be given priority and their mesh should be requested first.
    • 对于新曲面,更高优先级的优先级可能更高,特别是在扫描体验过程中。It may also be appropriate to give higher priority to new surfaces than to updated surfaces, especially during the scanning experience.
    • 为了限制处理成本,应用程序可能希望限制它们处理空间图面更新的速率。To limit processing costs, applications may wish to throttle the rate at which they process updates to spatial surfaces.
    • 可以推断出对空间图面所做的更改很小,例如,如果图面的边界很小,则更新可能不太重要,无法处理。It may be possible to infer that changes to a spatial surface are minor, for example if the bounds of the surface are small, in which case the update may not be important enough to process.
    • 对于用户感兴趣的当前区域之外的空间表面的更新,可能会完全忽略,但是在这种情况下,修改 surface 观察器使用的空间边界卷可能更有效。Updates to spatial surfaces outside the current region of interest of the user may be ignored entirely, though in this case it may be more efficient to modify the spatial bounding volumes in use by the surface observer.
  • 对于删除的空间表面,是否应丢弃网格?For removed spatial surfaces, should mesh be discarded?
    • 通常应为删除的空间图面立即丢弃网格,使全息图封闭保持正确。Generally mesh should be discarded immediately for removed spatial surfaces, so that hologram occlusion remains correct.
    • 但是,如果应用程序有理由相信某个空间图面很快就会重新出现 (基于用户体验) 的设计,则保留该网格比丢弃其网格并稍后重新创建它可能更有效。However, if the application has reason to believe that a spatial surface will reappear shortly (based upon the design of the user experience), then it may be more efficient to keep it than to discard its mesh and recreate it again later.
    • 如果应用程序正在生成用户环境的大规模模型,则可能根本不想丢弃任何网格。If the application is building a large-scale model of the user's environment, then it may not wish to discard any meshes at all. 不过,它仍需要限制资源使用情况,可能通过将网格后台打印到磁盘,因为空间图面消失。It will still need to limit resource usage though, possibly by spooling meshes to disk as spatial surfaces disappear.
    • 在空间表面生成过程中,某些相对罕见的事件可能会导致空间图面替换为类似位置但 Id 不同的新空间图面。Some relatively rare events during spatial surface generation can cause spatial surfaces to be replaced by new spatial surfaces in a similar location but with different IDs. 因此,选择不放弃已移除表面的应用程序应注意不会有多个高度重叠的空间表面,网格覆盖同一位置。So, applications that choose not to discard a removed surface should take care not to end up with multiple highly overlapped spatial surfaces meshes covering the same location.
  • 是否应为任何其他空间图面丢弃网格?Should mesh be discarded for any other spatial surfaces?
    • 即使存在空间图面,如果它对用户体验不再有用,则应将其丢弃。Even while a spatial surface exists, if it's no longer useful to the user's experience then it should be discarded. 例如,如果应用程序 "将门口的另一侧的空间替换为备用虚拟空间,则该房间中的空间表面将不再重要。For example, if the application 'replaces' the room on the other side of a doorway with an alternate virtual space then the spatial surfaces in that room no longer matter.

下面是使用空间和时态滞后的示例网状缓存策略:Here's an example mesh caching strategy, using spatial and temporal hysteresis:

  • 假设有一个应用程序,该应用程序想要在观看和四处浏览时使用在用户的 "注视" 周围的 "截锥" 空间量。Consider an application that wishes to use a frustum-shaped spatial volume of interest that follows the user's gaze as they look around and walk around.
  • 空间表面可能会暂时从此卷中消失,这只是因为用户看起来远离图面或其他步骤 .。。仅供稍后再次查找或更近一段时间。A spatial surface may disappear temporarily from this volume simply because the user looks away from the surface or steps further away from it... only to look back or moves closer again a moment later. 在这种情况下,放弃并重新创建此图面的网格表示许多冗余 processings。In this case, discarding and re-creating the mesh for this surface represents many redundant processings.
  • 为了减少处理的更改的数量,应用程序将使用两个空间图面观察器,其中一个包含在另一个中。To reduce the number of changes processed, the application uses two spatial surface observers, one contained within the other. 较大的卷为球状,并跟随用户 "延迟";它仅在必要时才会移动,以确保其中心在用户的2.0 米以内。The larger volume is spherical and follows the user 'lazily'; it only moves when necessary to ensure that its center is within 2.0 meters of the user.
  • 新的和更新的空间 surface 网格始终从较小的内部面观察程序进行处理,但会缓存网格,直到它们从较大的外部表面观察器消失。New and updated spatial surface meshes are always processed from the smaller inner surface observer, but meshes are cached until they disappear from the larger outer surface observer. 这使应用程序可以避免因本地用户移动而处理许多冗余更改。This allows the application to avoid processing many redundant changes because of local user movement.
  • 由于由于跟踪丢失,空间图面也可能暂时消失,因此,应用程序也会在跟踪丢失期间放弃删除的空间图面。Since a spatial surface may also disappear temporarily because of tracking loss, the application also defers discarding removed spatial surfaces during tracking loss.
  • 通常情况下,应用程序应评估减少的更新处理和增加的内存使用情况之间的折衷,以确定其理想的缓存策略。In general, an application should evaluate the tradeoff between reduced update processing and increased memory usage to determine its ideal caching strategy.

渲染Rendering

空间映射网格通常用于呈现的主要方式有三种:There are three primary ways in which spatial mapping meshes tend to be used for rendering:

  • 用于表面可视化For surface visualization
    • 直接直观显示空间图非常有用。It's often useful to visualize spatial surfaces directly. 例如,将 "shadows" 从对象强制转换为空间图面时,可以向用户提供更好的视觉反馈。For example, casting 'shadows' from objects onto spatial surfaces can provide helpful visual feedback to the user while they're placing holograms on surfaces.
    • 需要注意的一点是,空间网格不同于三维艺术家可能创建的网格类型。One thing to bear in mind is that spatial meshes are different to the kind of meshes that a 3D artist might create. 三角形拓扑不会作为用户创建的拓扑 "干净",并且网格会受到 各种错误的影响。The triangle topology won't be as 'clean' as human-created topology, and the mesh will suffer from various errors.
    • 若要创建令人满意的视觉对象美观,可能需要进行一些 网格处理,例如填充孔或平滑曲面法。To create a pleasing visual aesthetic, you may want to do some mesh processing, for example to fill holes or smooth surface normals. 您还可能希望使用着色器将艺术家设计的纹理投影到网格上,而不是直接可视化网格拓扑和法线。You may also wish to use a shader to project artist-designed textures onto your mesh instead of directly visualizing mesh topology and normals.
  • 对于真实的图面后面的 occluding 全息影像For occluding holograms behind real-world surfaces
    • 空间图面可以在仅深度传递中呈现,这只会影响 深度缓冲区 并且不会影响颜色呈现器目标。Spatial surfaces can be rendered in a depth-only pass, which only affects the depth buffer and doesn't affect color render targets.
    • 这 primes 了深度缓冲区,遮蔽随后在空间图面后面呈现全息影像。This primes the depth buffer to occlude subsequently rendered holograms behind spatial surfaces. 最准确的全息影像封闭增强了全息影像在用户的物理空间内确实存在的意义。Accurate occlusion of holograms enhances the sense that holograms really exist within the user's physical space.
    • 若要启用仅深度渲染,请更新混合状态,将所有颜色渲染目标的 RenderTargetWriteMask 设置为零。To enable depth-only rendering, update your blend state to set the RenderTargetWriteMask to zero for all color render targets.
  • 用于修改真实环境中全息图封闭像素的外观For modifying the appearance of holograms occluded by real-world surfaces
    • 当封闭像素时,通常会隐藏呈现的几何图形。Normally rendered geometry is hidden when it's occluded. 这是通过将 深度模具状态 中的深度函数设置为 "小于或等于" 来实现的,这会使几何仅在 离相机更近 的位置(而不是所有先前呈现的几何)可见。This is achieved by setting the depth function in your depth-stencil state to "less than or equal", which causes geometry to be visible only where it's closer to the camera than all previously rendered geometry.
    • 但是,在某些情况下,即使在封闭像素时也能看到某些几何,并在封闭像素作为向用户提供视觉反馈时修改其外观。However, it may be useful to keep certain geometry visible even when it's occluded, and to modify its appearance when occluded as a way of providing visual feedback to the user. 例如,这允许应用程序向用户显示对象的位置,同时使其清楚地显示在实际表面。For example, this allows the application to show the user the location of an object while making it clear that is behind a real-world surface.
    • 若要实现此目的,请使用另一个创建所需 "封闭像素" 外观的着色器第二次渲染几何。To achieve this, render the geometry a second time with a different shader that creates the desired 'occluded' appearance. 在第二次渲染几何之前,对 深度模具状态进行两次更改。Before rendering the geometry for the second time, make two changes to your depth-stencil state. 首先,将 "深度函数" 设置为 "大于或等于",以便仅在 其从相机中看到 的位置比之前呈现的所有几何都可见。First, set the depth function to "greater than or equal" so that the geometry will be visible only where it's further from the camera than all previously rendered geometry. 其次,将 DepthWriteMask 设置为零,以便不会修改深度缓冲区 (深度缓冲区应继续表示距离相机) 最近 的几何深度。Second, set the DepthWriteMask to zero, so that the depth buffer won't be modified (the depth buffer should continue to represent the depth of the geometry closest to the camera).

在呈现空间映射网格时,性能是一个重要的考虑因素。Performance is an important concern when rendering spatial mapping meshes. 下面是一些特定于呈现空间映射网格的呈现性能技巧:Here are some rendering performance techniques specific to rendering spatial mapping meshes:

  • 调整三角形密度Adjust triangle density
    • 从 surface 观察者请求空间表面网格时,请求可满足您需求的最小的三角形网格密度。When requesting spatial surface meshes from your surface observer, request the lowest density of triangle meshes that will suffice for your needs.
    • 在图面上改变三角密度可能有意义,具体取决于表面与用户的距离以及与用户体验的相关性。It may make sense to vary triangle density on a surface by surface basis, depending on the surface's distance from the user, and its relevance to the user experience.
    • 减少三角形计数将减少 GPU 上的内存使用量和顶点处理开销,但不会影响像素处理开销。Reducing triangle counts will reduce memory usage and vertex processing costs on the GPU, though it won't affect pixel processing costs.
  • 使用截锥剔除Use frustum culling
    • 截锥剔除会跳过无法看见的绘图对象,因为它们在当前显示的 "截锥" 之外。Frustum culling skips drawing objects that cannot be seen because they are outside the current display frustum. 这同时降低了 CPU 和 GPU 处理成本。This reduces both CPU and GPU processing costs.
    • 由于在每个网格基础上执行剔除,空间表面可能会很大,因此将每个空间曲面网格分割为较小的区块可能会导致更有效的剔除 (,因为) 会呈现较少的屏幕外三角形。Since culling is performed on a per-mesh basis and spatial surfaces can be large, breaking each spatial surface mesh into smaller chunks may result in more efficient culling (in that fewer offscreen triangles are rendered). 不过,有一种折衷:您拥有的网格越多,您必须进行的绘图调用越多,这会增加 CPU 开销。There's a tradeoff, however; the more meshes you have, the more draw calls you must make, which can increase CPU costs. 在极端情况下,如截锥剔除计算,甚至可能有可度量的 CPU 开销。In an extreme case, the frustum culling calculations themselves could even have a measurable CPU cost.
  • 调整呈现顺序Adjust rendering order
    • 空间表面很大,因为它们代表用户在周围的整个环境。Spatial surfaces tend to be large, because they represent the user's entire environment surrounding them. GPU 上的像素处理开销可能很高,尤其是在有多层可见的几何图形 (包括空间表面和其他全息影像) 的情况下。Pixel processing costs on the GPU can be high, especially in cases where there's more than one layer of visible geometry (including both spatial surfaces and other holograms). 在这种情况下,离用户最近的层会进一步 occluding 任何层,因此,在呈现这些更远距离的层所用的任何 GPU 时间都将浪费下来。In this case, the layer nearest to the user will be occluding any layers further away, so any GPU time spent rendering those more distant layers is wasted.
    • 若要减少 GPU 上的这一冗余工作,这有助于在最接近的情况下,以从上到下的顺序呈现不透明的图面, (上一次) 。To reduce this redundant work on the GPU, it helps to render opaque surfaces in front-to-back order (closer ones first, more distant ones last). "不透明" 是指在您的 深度模具状态中将 DepthWriteMask 设置为一个的表面。By 'opaque' we mean surfaces for which the DepthWriteMask is set to one in your depth-stencil state. 当呈现最近的图面时,它们将为深度缓冲区的质数,以便 GPU 上的像素处理器能够有效地跳过更远的图面。When the nearest surfaces are rendered, they'll prime the depth buffer so that more distant surfaces are efficiently skipped by the pixel processor on the GPU.

网格处理Mesh Processing

应用程序可能希望对空间 surface 网格执行 各种操作 以满足其需求。An application may want to do various operations on spatial surface meshes to suit its needs. 每个空间 surface 网格提供的索引和顶点数据与用于在所有新式渲染 Api 中呈现三角形网格的 顶点和索引缓冲区 使用的布局相同。The index and vertex data provided with each spatial surface mesh uses the same familiar layout as the vertex and index buffers that are used for rendering triangle meshes in all modern rendering APIs. 不过,有一个关键事实需要注意的是,空间映射三角形具有 前后的缠绕顺序However, one key fact to be aware of is that spatial mapping triangles have a front-clockwise winding order. 每个三角形都用网格的索引缓冲区中的三个顶点索引表示,当从 正面 查看三角形时,这些索引会按 顺时针 顺序标识三角形的顶点。Each triangle is represented by three vertex indices in the mesh's index buffer and these indices will identify the triangle's vertices in a clockwise order, when the triangle is viewed from the front side. 空间 surface 网格的正面 (或外部) 与您预期在实际表面的正面 () 可见。The front side (or outside) of spatial surface meshes corresponds as you would expect to the front (visible) side of real world surfaces.

如果 surface 观察器提供的益处三角形密度仍不足粗,则应用程序应仅执行网格简化,这种工作的计算成本非常高,且已由运行时执行以生成各种提供的详细信息级别。Applications should only do mesh simplification if the coarsest triangle density provided by the surface observer is still insufficiently coarse - this work is computationally expensive and already being performed by the runtime to generate the various provided levels of detail.

由于每个 surface 观察器都可以提供多个未连接的空间图面,因此某些应用程序可能希望彼此剪裁这些空间 surface 网格,然后将它们 zipper 在一起。Because each surface observer can provide multiple unconnected spatial surfaces, some applications may wish to clip these spatial surface meshes against each other, then zipper them together. 通常,剪裁步骤是必需的,因为附近的空间 surface 网格通常会稍微重叠。In general, the clipping step is required, as nearby spatial surface meshes often overlap slightly.

Raycasting 和冲突Raycasting and Collision

为了使物理学 API (例如 Havok) 为应用程序提供具有 raycasting 和冲突功能的空间图面,应用程序必须向物理学 api 提供空间 surface 网格。In order for a physics API (such as Havok) to provide an application with raycasting and collision functionality for spatial surfaces, the application must provide spatial surface meshes to the physics API. 用于物理学的网格通常具有以下属性:Meshes used for physics often have the following properties:

  • 它们只包含少量三角形。They contain only small numbers of triangles. 物理运算比呈现操作要多得多。Physics operations are more computationally intensive than rendering operations.
  • 它们是 "水紧的"。They're 'water-tight'. 设计为坚实的表面不应具有小孔;即使洞太小,也可能会导致问题。Surfaces intended to be solid shouldn't have small holes in them; even holes too small to be visible can cause problems.
  • 它们将转换为凸 hulls。They're converted into convex hulls. 凸 hulls 有几个多边形,并且没有孔,它们的计算效率比原始三角形网格要高得多。Convex hulls have few polygons and are free of holes, and they're much more computationally efficient to process than raw triangle meshes.

在对空间表面进行 raycasts 时,请记住,这些图面通常是复杂的、杂乱的形状,几乎不会有杂乱的细节,就像您的办公桌一样!When doing raycasts against spatial surfaces, bear in mind that these surfaces are often complex, cluttered shapes full of messy little details - just like your desk! 这意味着,单个 raycast 通常不能为您给出有关图面的形状和附近的空白空间形状的足够信息。This means that a single raycast is often insufficient to give you enough information about the shape of the surface and the shape of the empty space near it. 通常,在一个较小的区域中执行许多 raycasts,并使用聚合结果来更可靠地了解图面。It's usually a good idea to do many raycasts within a small area and to use the aggregate results to derive a more reliable understanding of the surface. 例如,使用平均 10 raycasts 来指导图面上的全息影像布局将产生比只使用单个 raycast 更流畅和更小的 "抖动" 结果。For example, using the average of 10 raycasts to guide hologram placement on a surface will yield a far smoother and less 'jittery' result that using just a single raycast.

不过,请记住,每个 raycast 都有一个较高的计算成本。However, bear in mind that each raycast can have a high computational cost. 根据您的使用情况,您应权衡额外的 raycasts (每个帧) 的计算成本,使其与 网格处理 的计算成本平滑和删除空间图面中的孔, (在空间网格更新) 时执行。Depending on your usage scenario, you should trade off the computational cost of extra raycasts (done every frame) against the computational cost of mesh processing to smooth and remove holes in spatial surfaces (done when spatial meshes are updated).

环境扫描体验The environment scanning experience

使用空间映射的每个应用程序都应考虑提供 "扫描体验";应用程序引导用户扫描应用程序正常运行所必需的表面所使用的过程。Each application that uses spatial mapping should consider providing a 'scanning experience'; the process through which the application guides the user to scan surfaces that are necessary for the application to function correctly.

扫描示例Example of scanning
扫描示例Example of scanning

此扫描体验的性质可能因每个应用程序的需求而有很大差别,但两个主要原则应指导其设计。The nature of this scanning experience can vary greatly depending upon each application's needs, but two main principles should guide its design.

首先, 清除与用户的通信是主要关注 点。Firstly, clear communication with the user is the primary concern. 用户应始终知道是否满足应用程序的要求。The user should always be aware of whether the application's requirements are being met. 当他们不能得到满足时,应立即向用户清楚地说明这是如此,并且应迅速采取适当的措施。When they aren't being met, it should be immediately clear to the user why this is so and they should be quickly led to take the appropriate action.

其次, 应用程序应尝试在效率和可靠性之间取得平衡Secondly, applications should attempt to strike a balance between efficiency and reliability. 如果可以 可靠地 执行此操作,则应用程序应自动分析空间映射数据以节省用户时间。When it's possible to do so reliably, applications should automatically analyze spatial mapping data to save the user time. 如果无法可靠地执行此操作,则应用程序应改为允许用户使用其所需的其他信息快速提供应用程序。When it isn't possible to do so reliably, applications should instead enable the user to quickly provide the application with the additional information it requires.

若要帮助设计正确的扫描体验,请考虑以下哪些可能适用于你的应用程序:To help design the right scanning experience, consider which of the following possibilities are applicable to your application:

  • 无扫描体验No scanning experience

    • 如果没有任何引导式扫描体验,应用程序可以正常工作;它将了解在自然用户移动过程中观察到的图面。An application may function perfectly without any guided scanning experience; it will learn about surfaces that are observed in the course of natural user movement.
    • 例如,允许用户在带有全息喷涂画图的表面上绘图的应用程序只需要知道当前对用户可见的表面。For example, an application that lets the user draw on surfaces with holographic spray paint requires knowledge only of the surfaces currently visible to the user.
    • 如果环境中的用户已使用 HoloLens 花费了大量时间,则可能会扫描该环境。The environment may be scanned already if it's one in which the user has already spent lots of time using the HoloLens.
    • 请记住,空间映射使用的相机只能在用户前面看到 3.1 m,因此,空间映射不会知道任何更远的图面,除非用户在过去从更远的距离观察到它们。Bear in mind however that the camera used by spatial mapping can only see 3.1 m in front of the user, so spatial mapping won't know about any more distant surfaces unless the user has observed them from a closer distance in the past.
    • 因此,用户了解已经扫描了哪些表面,应用程序应提供此效果的视觉反馈,例如,将虚拟阴影转换到扫描图面上可能会帮助用户在这些表面上放置全息影像。So the user understands which surfaces have been scanned, the application should provide visual feedback to this effect, for example casting virtual shadows onto scanned surfaces may help the user place holograms on those surfaces.
    • 对于这种情况,应将空间图面观察器的边界卷更新为主体锁定的 空间坐标系统,使其跟随用户。For this case, the spatial surface observer's bounding volumes should be updated each frame to a body-locked spatial coordinate system, so that they follow the user.
  • 查找合适的位置Find a suitable location

    • 应用程序可设计为在具有特定要求的位置使用。An application may be designed for use in a location with specific requirements.
    • 例如,应用程序可能需要用户附近的空白区域,以便他们能够安全地练习全息 kung-fu。For example, the application may require an empty area around the user so they can safely practice holographic kung-fu.
    • 应用程序应提前向用户传达任何特定要求,并通过清晰的视觉反馈来强化。Applications should communicate any specific requirements to the user up-front, and reinforce them with clear visual feedback.
    • 在此示例中,应用程序应可视化所需空区域的范围,并以可视方式突出显示此区域内任何不需要的对象的状态。In this example, the application should visualize the extent of the required empty area and visually highlight the presence of any undesired objects within this zone.
    • 对于这种情况,空间图面观察器的边界卷应在所选位置使用世界锁定的 空间坐标系统For this case, the spatial surface observer's bounding volumes should use a world-locked spatial coordinate system in the chosen location.
  • 查找图面的合适配置Find a suitable configuration of surfaces

    • 应用程序可能需要特定的图面配置,例如两个大型、平整、相反的壁,以创建全息的镜像厅。An application may require a specific configuration of surfaces, for example two large, flat, opposing walls to create a holographic hall of mirrors.
    • 在这种情况下,应用程序需要对空间映射提供的图面进行分析以检测适当的图面,并将用户定向到它们。In such cases, the application will need to analyze the surfaces provided by spatial mapping to detect suitable surfaces, and direct the user toward them.
    • 如果应用程序的表面分析不可靠,用户应具有回退选项。The user should have a fallback option if the application's surface analysis isn't reliable. 例如,如果应用程序错误地将门口标识为平整壁,则用户需要一种简单的方法来更正此错误。For example, if the application incorrectly identifies a doorway as a flat wall, the user needs a simple way to correct this error.
  • 扫描部分环境Scan part of the environment

    • 应用程序可能希望只捕获用户所指示的部分环境。An application may wish to only capture part of the environment, as directed by the user.
    • 例如,应用程序会扫描房间的某个部分,以使用户可以为想要销售的家具发布全息保密广告。For example, the application scans part of a room so the user may post a holographic classified ad for furniture they wish to sell.
    • 在这种情况下,应用程序应该捕获用户在扫描期间观察到的区域内的空间映射数据。In this case, the application should capture spatial mapping data within the regions observed by the user during their scan.
  • 扫描整个聊天室Scan the whole room

    • 应用程序可能需要扫描当前房间中的所有表面,包括用户之后的所有表面。An application may require a scan of all of the surfaces in the current room, including those behind the user.
    • 例如,游戏可能将用户作为 Gulliver 的角色,在 siege 中,从数百个微小 Lilliputians 接近所有方向。For example, a game may put the user in the role of Gulliver, under siege from hundreds of tiny Lilliputians approaching from all directions.
    • 在这种情况下,应用程序需要确定当前房间中已经扫描了多少个面,并指示用户的注视填充了重大间隙。In such cases, the application will need to determine how many of the surfaces in the current room have already been scanned, and direct the user's gaze to fill in significant gaps.
    • 此过程的关键是提供可视反馈,使用户清楚地知道尚未扫描哪些表面。The key to this process is providing visual feedback that makes it clear to the user which surfaces haven't yet been scanned. 例如,应用程序可以使用 基于距离的雾化 来以可视方式突出显示空间映射图面未涵盖的区域。The application could, for example, use distance-based fog to visually highlight regions that aren't covered by spatial mapping surfaces.
  • 获取环境的初始快照Take an initial snapshot of the environment

    • 在采用初始 "快照" 后,应用程序可能希望忽略环境中的所有更改。An application may wish to ignore all changes in the environment after taking an initial 'snapshot'.
    • 这可能适用于避免与环境中的初始状态紧密耦合的用户创建数据的中断。This may be appropriate to avoid disruption of user-created data that is tightly coupled to the initial state of the environment.
    • 在这种情况下,在扫描完成后,应用程序应在其初始状态下生成空间映射数据的副本。In this case, the application should make a copy of the spatial mapping data in its initial state once the scan is complete.
    • 如果在环境中仍有封闭像素,则应用程序应继续接收空间映射数据的更新。Applications should continue receiving updates to spatial mapping data if holograms are still to be correctly occluded by the environment.
    • 对于空间映射数据的持续更新,还允许对已发生的任何更改进行可视化,并向用户阐明环境以前和当前状态之间的差异。Continued updates to spatial mapping data also allow visualizing any changes that have occurred, clarifying to the user the differences between prior and present states of the environment.
  • 采用用户启动的环境快照Take user-initiated snapshots of the environment

    • 应用程序可能只希望在用户指示时对环境更改做出响应。An application may only wish to respond to environmental changes when instructed by the user.
    • 例如,用户可以通过在不同时刻捕获其姿势来创建多个 3D "statues"。For example, the user could create multiple 3D 'statues' of a friend by capturing their poses at different moments.
  • 允许用户更改环境Allow the user to change the environment

    • 应用程序可用于实时响应用户环境中所做的任何更改。An application may be designed to respond in real time to any changes made in the user's environment.
    • 例如,如果在另一侧发生全息部游戏,则用户绘制窗帘可能会触发 "场景变化"。For example, the user drawing a curtain could trigger 'scene change' for a holographic play taking place on the other side.
  • 指导用户避免空间映射数据中的错误Guide the user to avoid errors in the spatial mapping data

    • 应用程序在扫描其环境时,可能希望向用户提供指导。An application may wish to provide guidance to the user while they're scanning their environment.
    • 这可以帮助用户避免 空间映射数据中出现某些类型的错误,例如,远离 sunlit windows 或镜像。This can help the user to avoid certain kinds of errors in the spatial mapping data, for example by staying away from sunlit windows or mirrors.

需要注意的一个额外细节是,空间映射数据的 "范围" 不受限制。One extra detail to be aware of is that the 'range' of spatial mapping data isn't unlimited. 虽然空间映射确实生成了大量空间的永久数据库,但它只会使该数据可用于用户周围大小限制较小的应用程序。While spatial mapping does build a permanent database of large spaces, it only makes that data available to applications in a 'bubble' of limited size around the user. 如果从较长的 corridor 开始开始,并从一开始就开始走出足够的距离,最终空间表面会消失。If you start at the beginning of a long corridor and walk far enough away from the start, then eventually the spatial surfaces back at the beginning will disappear. 在应用程序从可用空间映射数据中消失后,可以通过在应用程序中缓存这些图面来缓解这种情况。You can mitigate this by caching those surfaces in your application after they've disappeared from the available spatial mapping data.

网格处理Mesh processing

它可能有助于检测图面中的常见错误类型,并根据需要对空间映射数据进行筛选、删除或修改。It may help to detect common types of errors in surfaces and to filter, remove or modify the spatial mapping data as appropriate.

请记住,空间映射数据应尽可能忠实到实际的表面,因此,任何正在进行的处理都将进一步从 "真实" 改变你的曲面。Bear in mind that spatial mapping data is intended to be as faithful as possible to real-world surfaces, so any processing you apply risks shifting your surfaces further from the 'truth'.

下面是一些可能有用的不同类型网格处理的示例:Here are some examples of different types of mesh processing that you may find useful:

  • 孔洞填充Hole filling

    • 如果导致暗材料的小对象无法扫描,它将在周围表面上留下一个孔。If a small object made of a dark material fails to scan, it will leave a hole in the surrounding surface.
    • 孔洞会影响封闭:在被视为不透明的实际表面,可以通过 "通过" 洞来查看全息影像。Holes affect occlusion: holograms can be seen 'through' a hole in a supposedly opaque real-world surface.
    • 孔洞会影响 raycasts:如果使用 raycasts 帮助用户与表面交互,则这些光线可能不需要通过洞。Holes affect raycasts: if you're using raycasts to help users interact with surfaces, it may be undesirable for these rays to pass through holes. 一种缓解措施是使用多个 raycasts 的捆绑,涵盖适当大小的区域。One mitigation is to use a bundle of multiple raycasts covering an appropriately sized region. 这将允许你筛选 "离群离群" 结果,这样即使一个 raycast 通过一个小孔,聚合结果仍将有效。This will allow you to filter 'outlier' results, so that even if one raycast passes through a small hole, the aggregate result will still be valid. 但是,这种方法会产生计算成本。However, this approach comes at a computational cost.
    • 孔洞会影响物理学冲突:物理学模拟控制的对象可能会贯穿地面的孔,并会丢失。Holes affect physics collisions: an object controlled by physics simulation may drop through a hole in the floor and become lost.
    • 可以算法在 surface 网格中填充此类孔。It's possible to algorithmically fill such holes in the surface mesh. 但是,你需要调整算法,以便不会填充 "real" (如 windows 和两侧)。However, you'll need to tune your algorithm so that 'real holes' such as windows and doorways don't get filled in. 可能很难可靠地区分 "真实漏洞" 和 "虚部",因此需要尝试不同的试探法,如 "大小" 和 "边界形状"。It can be difficult to reliably differentiate 'real holes' from 'imaginary holes', so you'll need to experiment with different heuristics such as 'size' and 'boundary shape'.
  • 删除 HallucinationHallucination removal

    • 反射、明亮灯光和移动对象可能会使小延迟 "hallucinations" 浮动在中。Reflections, bright lights, and moving objects can leave small lingering 'hallucinations' floating in mid-air.
    • Hallucinations 影响封闭: Hallucinations 可能会成为深色形状,并将其移到上,并 occluding 其他全息影像。Hallucinations affect occlusion: hallucinations may become visible as dark shapes moving in front of and occluding other holograms.
    • Hallucinations 影响 raycasts:如果你使用 raycasts 帮助用户与表面交互,则这些光线可能会碰到 hallucination,而不是其背后的表面。Hallucinations affect raycasts: if you're using raycasts to help users interact with surfaces, these rays could hit a hallucination instead of the surface behind it. 与孔洞一样,一种缓解措施是使用许多 raycasts,而不是使用单个 raycast,但这会产生计算成本。As with holes, one mitigation is to use many raycasts instead of a single raycast, but again this will come at a computational cost.
    • Hallucinations 影响物理学冲突:物理学模拟控制的对象可能会停滞在 hallucination 的位置,并且无法在看似清晰的空间区域中移动。Hallucinations affect physics collisions: an object controlled by physics simulation may become stuck against a hallucination and be unable to move through a seemingly clear area of space.
    • 可以从 surface 网格筛选此类 hallucinations。It's possible to filter such hallucinations from the surface mesh. 但是,与孔洞一样,你需要调整算法,使真实的小对象(如灯具和门控点)不会被删除。However, as with holes, you'll need to tune your algorithm so that real small objects such as lamp-stands and door handles don't get removed.
  • 光滑Smoothing

    • 空间映射可能会返回与实际对应项相比显得非常粗糙或 "干扰" 的表面。Spatial mapping may return surfaces that appear to be rough or 'noisy' in comparison to their real-world counterparts.
    • 平滑度会影响物理学冲突:如果地面为粗糙,则物理模拟的高尔夫球可能不会以直线平滑地滚动。Smoothness affects physics collisions: if the floor is rough, a physically simulated golf ball may not roll smoothly across it in a straight line.
    • 平滑度会影响呈现:如果图面直接可视化,则视觉表面的法线会影响其外观并干扰 "干净" 的外观。Smoothness affects rendering: if a surface is visualized directly, rough surface normals can affect its appearance and disrupt a 'clean' look. 可以通过使用着色器中用于呈现图面的适当照明和纹理来缓解这种情况。It's possible to mitigate this by using appropriate lighting and textures in the shader that is used to render the surface.
    • 可以在 surface 网格中平滑粗糙度。It's possible to smooth out roughness in a surface mesh. 但是,这可能会使表面远离相应的实际表面。However, this may push the surface further away from the corresponding real-world surface. 维护近函件对于生成准确的全息图封闭非常重要,并使用户能够通过全息表面实现精确、可预测的交互。Maintaining a close correspondence is important to produce accurate hologram occlusion, and to enable users to achieve precise and predictable interactions with holographic surfaces.
    • 如果只需要修饰的更改,则它可能足以使顶点法线平滑,而不会改变顶点位置。If only a cosmetic change is required, it may be sufficient to smooth vertex normals without changing vertex positions.
  • 平面查找Plane finding

    • 应用程序可能希望在空间映射提供的图面上执行多种形式的分析。There are many forms of analysis that an application may wish to perform on the surfaces provided by spatial mapping.
    • 一个简单的示例是 "平面查找";确定边界,大多数平面区域。One simple example is 'plane finding'; identifying bounded, mostly planar regions of surfaces.
    • 平面区域可用作全息的工作图面,其中全息内容可由应用程序自动放置。Planar regions can be used as holographic work-surfaces, regions where holographic content can be automatically placed by the application.
    • 平面区域可以限制用户界面,指导用户与最适合其需求的图面进行交互。Planar regions can constrain the user interface, to guide users to interact with the surfaces that best suit their needs.
    • 在现实世界中,可以将平面区域用作可用于功能对象(如液晶屏、表格或白板)的全息物。Planar regions can be used as in the real world, for holographic counterparts to functional objects such as LCD screens, tables or whiteboards.
    • 平面区域可以定义播放区域,形成视频游戏级别的基础。Planar regions can define play areas, forming the basis of video game levels.
    • 平面区域可以帮助虚拟代理浏览现实世界,方法是确定真实人员可能会走到的地面区域。Planar regions can aid virtual agents to navigate the real world, by identifying the areas of floor that real people are likely to walk on.

原型制作和调试Prototyping and debugging

有用工具Useful tools

  • HoloLens 模拟器可用于使用空间映射开发应用程序,而无需访问物理 HoloLens。The HoloLens emulator can be used to develop applications using spatial mapping without access to a physical HoloLens. 使用该功能,您可以在真实环境中使用您的应用程序通常会使用的所有数据(包括 HoloLens 运动、空间坐标系统和空间映射网格)来模拟 HoloLens 上的实时会话。It allows you to simulate a live session on a HoloLens in a realistic environment, with all of the data your application would normally consume, including HoloLens motion, spatial coordinate systems, and spatial mapping meshes. 这可用于提供可靠的可重复输入,这对于调试问题和评估代码更改很有用。This can be used to provide reliable, repeatable input, which can be useful for debugging problems and evaluating changes to your code.
  • 若要重现方案,请从实时 HoloLens 捕获网络上的空间映射数据,然后将其保存到磁盘,并在以后的调试会话中重复使用。To reproduce a scenario, capture spatial mapping data over the network from a live HoloLens, then save it to disk and reuse it in later debugging sessions.
  • Windows 设备门户三维视图提供了一种方法,用于查看当前通过空间映射系统提供的所有空间图面。The Windows device portal 3D view provides a way to see all of the spatial surfaces currently available via the spatial mapping system. 这为应用程序内的空间曲面提供了比较基础;例如,您可以很容易地判断是否有任何空间图面丢失或显示在错误的位置。This provides a basis of comparison for the spatial surfaces inside your application; for example, you can easily tell if any spatial surfaces are missing or are being displayed in the wrong place.

一般原型设计指南General prototyping guidance

  • 由于空间映射数据中的 错误 可能会对用户体验造成很大的影响,因此建议在各种环境中测试应用程序。Because errors in the spatial mapping data may strongly affect your user's experience, we recommend that you test your application in a wide variety of environments.
  • 不要以始终在同一位置(例如在办公桌)上进行测试的习惯进行捕获。Don't get trapped in the habit of always testing in the same location, for example at your desk. 请确保在不同位置、形状、大小和材料的各种曲面上进行测试。Make sure to test on various surfaces of different positions, shapes, sizes, and materials.
  • 同样,尽管综合数据或记录的数据对于调试非常有用,但不太依赖于相同的几个测试用例。Similarly, while synthetic or recorded data can be useful for debugging, don't become too reliant upon the same few test cases. 这可能会延迟查找更多测试会提前捕获的重要问题。This may delay finding important issues that more varied testing would have caught earlier.
  • 使用真实 (和理想的 uncoached) 用户进行测试是一个不错的做法,因为它们不能使用 HoloLens 或您的应用程序的方式与您的应用程序完全相同。It's a good idea to perform testing with real (and ideally uncoached) users, because they may not use the HoloLens or your application in exactly the same way that you do. 事实上,这可能会让您感到他人的行为、知识和假设有多多。In fact, it may surprise you how divergent people's behavior, knowledge, and assumptions can be!

疑难解答Troubleshooting

  • 为了正确地面向 surface 网格,每个 GameObject 都需要处于活动状态,然后才能将其发送到 SurfaceObserver 来构造其网格。In order for the surface meshes to be orientated correctly, each GameObject needs to be active before it's sent to the SurfaceObserver to have its mesh constructed. 否则,网格将显示在您的空间中,但会旋转为古怪度。Otherwise, the meshes will show up in your space but rotated at weird angles.
  • 需要将运行与 SurfaceObserver 通信的脚本的 GameObject 设置为源。The GameObject that runs the script that communicates with the SurfaceObserver needs to be set to the origin. 否则,你创建并发送到 SurfaceObserver 的所有 Gameobject 都具有与父游戏对象的偏移量相等的偏移量。Otherwise, all of GameObjects that you create and send to the SurfaceObserver to have their meshes constructed will have an offset equal to the offset of the Parent Game Object. 这会使你的网格显示多个米,这使得调试正在进行的操作变得很困难。This can make your meshes show up several meters away, which makes it hard to debug what is going on.

另请参阅See also