使用手指向和提交Point and commit with hands

光标

“用手指向和提交”是一种输入模型,它使用户能够对无法触及的 2D 和 3D 内容进行定位、选择和操作。Point and commit with hands is an input model that lets users target, select, and manipulate out of reach 2D and 3D content. 这种“远程”交互技术是混合现实所独有的,因为人类不会自然地以这种方式与现实世界交互。This "far" interaction technique is unique to mixed reality because humans don't naturally interact with the real world that way. 例如,在超级英雄电影《X 战警》中,角色 Magneto 能够用手操控远处的对象。For example, in the super hero movie, X-Men, the character Magneto can manipulate far objects in the distance with his hands. 这是人类在现实中无法做到的。This isn't something humans can do in reality. 在 HoloLens (AR) 和混合现实 (MR) 中,我们赋予了用户这种神奇的力量,打破了现实世界的物理约束。In both HoloLens (AR) and Mixed Reality (MR), we equip users with this magical power to break the physical constraint of the real world. 这不仅能让用户享受到愉悦的全息体验,还能让互动更加有效、高效。Not only is it a fun holographic experience, but it also makes user interactions more effective and efficient.

设备支持Device support

输入模型Input model HoloLens(第 1 代)HoloLens (1st gen) HoloLens 2HoloLens 2 沉浸式头戴显示设备Immersive headsets
使用手指向和提交Point and commit with hands ❌ 不支持❌ Not supported ✔️ 推荐✔️ Recommended ✔️ 推荐✔️ Recommended

“用手指向和提交”是使用新的铰接式手跟踪系统的新功能之一。"Point and commit with hands" is one of the new features that use the new articulated hand-tracking system. 该输入模型也是沉浸式头戴显示设备上通过使用运动控制器实现的主要输入模式。This input model is also the primary input model on immersive headsets by using motion controllers.



手部射线Hand rays

在 HoloLens 2 上,我们创造了一种从用户手掌中心射出的手部射线。On HoloLens 2, we created a hand ray that shoots out from the center of the user's palm. 该射线被视为手的延伸。This ray is treated as an extension of the hand. 圆环形光标连接到射线的末端,以指示射线与目标对象相交的位置。A donut-shaped cursor is attached to the end of the ray to indicate the location where the ray intersects with a target object. 然后光标所指向的对象可以接收来自手的手势命令。The object that the cursor lands on can then receive gestural commands from the hand.

该基本手势命令通过使用拇指和食指进行隔空敲击动作来触发。This basic gestural command is triggered by using the thumb and index finger to do the air-tap action. 通过使用手射线指向和隔空敲击以提交,用户可以激活某个按钮或超链接。By using the hand ray to point and air tap to commit, users can activate a button or a hyperlink. 借助更多的复合手势,用户可以在 Web 内容中导航,并从远处操纵 3D 对象。With more composite gestures, users can navigating web content and manipulating 3D objects from a distance. 手部射线的视觉设计也应对这些指向和提交状态做出反应,如下所述:The visual design of the hand ray should also react to these point and commit states, as described and shown below:

手部射线指向hand rays pointing
“指向”状态Pointing state
在指向状态下,射线是虚线,光标是圆环形状 。In the pointing state, the ray is a dash line and the cursor is a donut shape.

手部射线提交hand rays commit
“提交”状态Commit state
在提交状态下,光线变为实线,光标缩小为点 。In the commit state, the ray turns into a solid line and the cursor shrinks to a dot.



在远近之间转换Transition between near and far

我们没有使用“食指指向”这样的特定手势来引导射线,而是将射线设计为从用户手掌中心发出。Instead of using specific gestures like "pointing with index finger" to direct the ray, we designed the ray to comout out from the center of the users' palm. 这样,我们解放并留出了五根手指来做更多的操控性手势,如捏和抓。This way, we've released and reserved the Five Fingers for more manipulative gestures like pinch and grab. 通过这种设计,我们只创建了一个心理模型 - 同一组手势用于远交互和近交互。With this design, we create only one mental model - the same set of hand gestures are used for both near and far interaction. 可以使用相同的抓取手势来操纵不同距离的对象。You can use the same grab gesture to manipulate objects at different distances. 射线的调用是自动的,并且基于邻近度,如下所述:The invocation of the rays is automatic and proximity-based as follows:

近操作Near manipulation
近操作Near manipulation
当对象在手臂长度范围(大约 50 cm)内时,射线会自动关闭,支持近距交互。When an object is within arm's length (roughly 50 cm), the rays are turned off automatically, encouraging near interaction.

远操作Far manipulation
远操作Far manipulation
当对象距离超过 50 cm 时,将启用射线。When the object is farther than 50 cm, the rays are turned on. 转换应该顺利无缝。The transition should be smooth and seamless.



2D 平板电脑交互2D slate interaction

2D 盖板是装有 Web 浏览器等 2D 应用内容的全息容器。A 2D slate is a holographic container hosting 2D app contents, such as a web browser. 与 2D 平板电脑进行远距离交互的设计理念是使用手部射线进行瞄准,并隔空敲击以进行选择。The design concept for far interacting with a 2D slate is to use hand rays to target and air tap to select. 在用手部射线瞄准目标后,用户可以通过隔空敲击来触发超链接或按钮。After targeting with a hand ray, users can air tap to trigger a hyperlink or a button. 他们可以用一只手“隔空敲击并拖动”来上下滚动盖板内容。They can use one hand to "air tap and drag" to scroll slate content up and down. 使用两只手进行隔空敲击和拖动的相对运动可以放大和缩小平板电脑内容。The relative motion of using two hands to air tap and drag can zoom in and out the slate content.

在角落和边缘处瞄准手部射线可显示最接近的可视操作元素。Targeting the hand ray at the corners and edges reveals the closest manipulation affordance. 通过“抓取和拖动”操作视觉元素,用户可通过边角视觉元素进行统一缩放,然后通过边缘视觉元素重排盖板。By "grab and drag" manipulation affordances, users can do uniform scaling through the corner affordances, and can reflow the slate via the edge affordances. 用户可以通过抓取并拖动 2D 盖板顶部的全息条来移动整个盖板。Grabbing and dragging the holobar at the top of the 2D slate lets users move the entire slate.

2D 盖板交互单击2d slate interaction click
单击Click

2D 盖板交互滚动2d slate interaction scroll
滚动Scroll

2D 盖板交互缩放2d slate interaction zoom
缩放Zoom


操作 2D 盖板For manipulating the 2D slate

  • 用户在角落或边缘处指向手部射线可显示最接近的可视操作元素。Users point the hand ray at the corners or edges to reveal the closest manipulation affordance.
  • 通过对视觉元素应用操作手势,用户可通过边角视觉元素进行统一缩放,然后通过边缘视觉元素重排盖板。By applying a manipulation gesture on the affordance, users can do uniform scaling through the corner affordance, and can reflow the slate via the edge affordance.
  • 通过在 2D 盖板顶部的全息条上应用操纵手势,用户可以移动整个盖板。By applying a manipulation gesture on the holobar at the top of the 2D slate, users can move the entire slate.


3D 对象操作3D object manipulation

在直接操作中,用户可以通过两种方式操纵 3D 对象:基于视觉元素的操作和非基于视觉元素的操作。In direct manipulation, there are two ways for users to manipulate 3D objects: affordance-based manipulation and non-affordance based manipulation. 在指向和提交模型中,用户能够通过手部射线实现完全相同的任务。In the point and commit model, users can achieve exactly the same tasks through the hand rays. 不需要额外的学习。No extra learning is needed.

基于可视操作元素的操作Affordance-based manipulation

用户使用手部射线指向并显示边框和可视操作元素。Users use hand rays to point and reveal the bounding box and manipulation affordances. 用户可在边框上应用操作手势来移动整个对象,在边缘视觉元素上应用操作手势来旋转对象,以及在边角视觉元素上应用操作手势以进行统一的缩放。Users can apply the manipulation gesture on the bounding box to move the whole object, on the edge affordances to rotate, and on the corner affordances to scale uniformly.

3D 对象操作远移动3d object manipulation far move
移动Move

3D 对象操作远旋转3d object manipulation far rotate
旋转Rotate

3D 对象操作远缩放3d object manipulation far scale
缩放Scale

非基于视觉元素的操作Non-affordance-based manipulation

用户用手部射线指向以显示边界框,然后直接对其应用操作手势。Users point with hand rays to reveal the bounding box then directly apply manipulation gestures on it. 如果用一只手进行操作,对象的平移和旋转将与手的移动和方向相关联。With one hand, the translation and rotation of the object are associated to motion and orientation of the hand. 如果用两只手进行操作,用户可以根据两只手的相对运动来平移、缩放和旋转对象。With two hands, users can translate, scale, and rotate it according to relative motions of two hands.



本能手势Instinctual gestures

指向和提交的本能手势的概念类似于用手直接操作的概念。The concept of instinctual gestures for point and commit is similar to that for direct manipulation with hands. 用户在 3D 对象上执行的手势是由 UI 视觉元素的设计引导的。The gestures users do on a 3D object are guided by the design of UI affordances. 例如,一个小的控制点可能会促使用户用拇指和食指来捏,而用户可能希望用全部五根手指来抓一个较大的对象。For example, a small control point might motivate users to pinch with their thumb and index finger, while a user might want to use all Five Fingers to grab a larger object.

本能手势远的小型对象instinctual gestures far small object
小型对象Small object

本能手势远的中型对象instinctual gestures far medium object
中型对象Medium object

本能手势远的大型对象instinctual gestures far large object
大型对象Large object



手部与 6 DoF 控制器之间的对称设计Symmetric design between hands and 6 DoF controller

远程交互的指向和提交概念最初是为混合现实门户 (MRP) 创建和定义的。The concept of point and commit for far interaction was created and defined for the Mixed Reality Portal (MRP). 在此场景下,用户戴上沉浸式头戴显示设备,通过运动控制器与 3D 对象进行交互。In this scenario, a user wears an immersive headset and interacts with 3D objects via motion controllers. 运动控制器射出光线来指向和操纵远处的对象。The motion controllers shoot out rays for pointing and manipulating far objects. 控制器上有用于进一步提交不同操作的按钮。There are buttons on the controllers for further committing different actions. 我们应用射线的交互模式,并将其与双手关联。We apply the interaction model of rays and attached them to both hands. 借助这种对称设计,熟悉 MRP 的用户在使用 HoloLens 2 时不需要学习另一种交互模式来进行远距离指向和操纵,反之亦然。With this symmetric design, users who are familiar with MRP won't need to learn another interaction model for far pointing and manipulation when they use HoloLens 2, and the other way around.

带控制器的射线的对称设计symmetric design for rays with controllers
控制器射线Controller rays

带手的射线的对称设计symmetric design for rays with hands
手部射线Hand rays



Unity 的 MRTK(混合现实工具包)中的手部射线Hand ray in MRTK (Mixed Reality Toolkit) for Unity

默认情况下,MRTK 提供一个手部射线 prefab (DefaultControllerPointer.prefab),其视觉状态与 shell 的系统手部射线相同。By default, MRTK provides a hand ray prefab (DefaultControllerPointer.prefab) which has the same visual state as the shell's system hand ray. 它分配在 MRTK 的输入配置文件的“指针”下。It's assigned in MRTK's Input profile, under Pointers. 在沉浸式头戴显示设备中,相同的射线会用于运动控制器。In an immersive headset, the same rays are used for the motion controllers.


另请参阅See also