HoloLens 2 中的眼动跟踪Eye tracking on HoloLens 2

MRTK 中的眼睛跟踪演示

HoloLens 2 允许开发人员使用有关用户正在查看的内容的信息,从而实现全新的上下文和人工理解。HoloLens 2 allows for a new level of context and human understanding within the holographic experience by providing developers with the ability to use information about what the user is looking at. 本页介绍了开发人员如何从各种用例的目视跟踪中获益,并介绍了在设计基于目视的用户交互时应了解的内容。This page explains how developers can benefit from eye tracking for various use cases, and what to look for when designing eye-gaze-based user interactions.

目视跟踪 API 的设计目的在于用户的隐私,避免传递任何可识别信息,尤其是任何生物识别。Eye tracking API has been designed with a user’s privacy in mind, avoiding passing any identifiable information, particularly any biometrics. 对于支持目视跟踪的应用程序,用户需要授予应用程序权限才能使用目视跟踪信息。For eye-tracking capable applications, the user needs to grant app permission to use eye tracking information.

设备支持Device support

功能Feature HoloLens(第 1 代)HoloLens (1st gen) HoloLens 2HoloLens 2 沉浸式头戴显示设备Immersive headsets
眼睛-注视Eye-gaze ✔️✔️

校准Calibration

为了使目视跟踪能准确地工作,每个用户都需要经历 跟踪用户校准 ,用户必须查看一组全息目标。For eye tracking to work accurately, each user is required to go through an eye tracking user calibration for which the user has to look at a set of holographic targets. 这允许设备调整系统,以便为用户提供更舒适和更高的质量查看体验,并确保同时进行准确的目视跟踪。This allows the device to adjust the system for a more comfortable and higher quality viewing experience for the user and to ensure accurate eye tracking at the same time.

目视跟踪应该适用于大多数用户,但在极少数情况下,用户无法成功校准。Eye tracking should work for most users, but there are rare cases in which a user can't calibrate successfully. 校准可能因多种原因而失败,其中包括但不限于:Calibration might fail for various reasons, including but not limited to:

  • 用户以前选择退出校准过程The user previously opted out of the calibration process
  • 用户已分散注意力,未遵循校准目标The user got distracted and didn't follow the calibration targets
  • 用户具有某些类型的 contact 重用功能区和眼镜,系统尚不支持该系统The user has certain types of contact lenses and glasses, which the system doesn't yet support
  • 用户具有某些眼睛 physiology、目视状况或状态外科,系统尚不支持该系统The user has certain eye physiology, eye conditions or had eye surgery, which the system doesn't yet support
  • 外部因素抑制可靠的眼睛跟踪,如在眼睛正面的眼睛上出现污迹,如污迹面板或眼镜、强烈直接阳光和 occlusionsExternal factors inhibiting reliable eye tracking such as smudges on the HoloLens visor or eyeglasses, intense direct sunlight, and occlusions due to hair in front of the eyes

开发人员应确保向用户提供对其视觉跟踪数据可能不可用 (无法成功) 的用户的充分支持。Developers should make sure to provide adequate support for users for whom eye tracking data may not be available (who aren't able to calibrate successfully). 我们已在本页底部的部分中提供了有关后备解决方案的建议。We have provided recommendations for fallback solutions in the section at the bottom of this page.

若要了解有关校准的详细信息以及如何确保流畅的体验,请查看我们的 眼睛跟踪用户校准 页面。To learn more about the calibration and about how to ensure a smooth experience, check our eye tracking user calibration page.


可用的目视跟踪数据Available eye tracking data

在深入了解有关目视输入的特定用例的详细信息之前,我们想要简要指出 HoloLens 2 目视跟踪 API 提供的功能。Before going into detail about specific use cases for eye-gaze input, we want to briefly point out the capabilities that the HoloLens 2 Eye Tracking API provides. 开发人员可在大约 30 FPS (30) Hz 的情况上,通过一种眼睛 (眼睛和方向) 访问。Developers get access to a single eye-gaze ray (gaze origin and direction) at approximately 30 FPS (30 Hz). 有关如何访问目视跟踪数据的详细信息,请参阅我们的开发人员指南,了解如何使用 DirectX 中的眼睛 ,并看看 UnityFor more detailed information about how to access eye tracking data, refer to our developer guides for using eye-gaze in DirectX and eye-gaze in Unity.

围绕实际目标 (的视觉角度看,眼睛的眼睛约为1.5 度,请参阅下图) 。The predicted eye-gaze is approximately within 1.5 degrees in visual angle around the actual target (see the illustration below). 由于预期的轻微 imprecisions,开发人员应围绕这一下限值来规划一些利润率 (例如,2.0-3.0 度可能会导致更舒适的体验) 。As slight imprecisions are expected, developers should plan for some margin around this lower-bound value (for example, 2.0-3.0 degrees may result in a much more comfortable experience). 下面将详细介绍如何处理小目标的选择。We'll discuss how to address the selection of small targets in more detail below. 要准确运行眼动跟踪,每个用户需要完成眼动跟踪用户校准。For eye tracking to work accurately, each user is required to go through an eye tracking user calibration.

2 米远处的最佳目标大小Optimal target size at 2 meter distance
以2米距离为目标的最佳目标大小Optimal target size at a 2-meter distance


用例Use cases

眼动跟踪可让应用程序实时跟踪用户正在注视的位置。Eye tracking enables applications to track where the user is looking in real time. 以下用例描述了在混合现实中,在 HoloLens 2 上使用目视跟踪可能会发生的一些交互。The following use cases describe some interactions that are possible with eye tracking on HoloLens 2 in mixed reality. 这些用例不是全息 Shell 体验的一部分 (即,启动 HoloLens 2) 时显示的接口。These use cases aren't yet part of the Holographic Shell experience (that is, the interface that you see when you start up your HoloLens 2). 你可以在 混合现实工具包中试用其中一些功能,这为使用目视跟踪提供了几个有趣且功能强大的示例,例如,快速轻松地支持目视的目标选择,并根据用户所查看的内容自动滚动查看文本。You can try some of them in the Mixed Reality Toolkit, which provides several interesting and powerful examples for using eye tracking, such as quick and effortless eye-supported target selections, and automatically scrolling through text based on what the user looks at.

用户意图User intent

有关用户所在位置和内容的信息 为其他输入 提供了强大的上下文,如语音、双手和控制器。Information about where and what a user looks at provides a powerful context for other inputs, such as voice, hands, and controllers. 可在各种任务中使用此信息。This can be used for various tasks. 例如,这可以通过查看全息图并口述 "选择" (,在场景中快速轻松地 定位,同时查看 "注视" 和 "提交") 或 "put ...",然后查找用户要放置全息影像的位置,并显示 ".。。出现 "For example, this can range from quickly and effortlessly targeting across the scene by looking at a hologram and saying "select" (also see gaze and commit) or "put this...", then looking over to where the user wants to place the hologram and say "...there". 混合现实工具包 - 视线支持的目标选择混合现实工具包 - 视线支持的目标定位中可以找到相关示例。Examples for this can be found in Mixed Reality Toolkit - Eye-supported Target Selection and Mixed Reality Toolkit - Eye-supported Target Positioning.

此外,用户意向的示例可能包括使用用户查看的信息来增强使用所介绍的虚拟代理和交互式全息影像。Additionally, an example for user intent might include using information about what users look at to enhance engagement with embodied virtual agents and interactive holograms. 例如,根据当前查看的内容,虚拟代理可能会调整可用选项及其行为。For instance, virtual agents might adapt available options and their behavior, based on currently viewed content.

隐式操作Implicit actions

隐式操作的类别与用户意图密切相关。The category of implicit actions closely relates to user intent. 其思路是,全息影像或用户界面元素会以 instinctual 的方式做出反应,甚至可能不会像用户同时与系统交互,而是让系统和用户保持同步。例如, 基于目视的自动滚动 ,用户可以在其中读取长文本,在用户进入文本框底部时自动开始滚动,以使用户处于阅读流中,而不会抬起手指。The idea is that holograms or user interface elements react in an instinctual way that may not even feel like the user is interacting with the system at all, but rather that the system and the user are in sync. One example is eye-gaze-based auto scroll where the user can read a long text, which automatically starts scrolling once the user gets to the bottom of the textbox to keep the user in the flow of reading, without lifting a finger.
这种情况的一个关键方面是,滚动速度可适应用户的读取速度。A key aspect of this is that the scrolling speed adapts to the reading speed of the user. 另一个例子就是 受支持的目视缩放和平移 ,用户可以感觉到他或她所关注的内容完全接近。Another example is eye-supported zoom and pan where the user can feel like diving exactly toward what he or she's focused on. 触发和控制缩放速度可以通过语音或手写输入来控制,这对于向用户提供控制感受,同时避免被淹没非常重要。Triggering and controlling zoom speed can be controlled by voice or hand input, which is important for providing the user with the feeling of control while avoiding being overwhelmed. 下面将更详细地讨论这些设计注意事项。We'll talk about these design considerations in more detail below. 放大后,用户可以通过使用眼睛来平稳地关注街道的学习过程,以浏览其邻居。Once zoomed in, the user can smoothly follow, for example, the course of a street to explore his or her neighborhood by using their eye-gaze. 混合现实工具包 - 视线支持的导航示例中可以找到此类交互的演示。Demo examples for these types of interactions can be found in the Mixed Reality Toolkit - Eye-supported Navigation sample.

隐式操作 的其他用例可能包括:Other use cases for implicit actions may include:

  • 智能通知: 在你要查找的位置,厌恶的通知会出现问题?Smart notifications: Ever get annoyed by notifications popping up right where you're looking? 考虑到用户正在关注的内容,你可以通过从当前 gazing 用户的位置偏移通知来更好地进行此体验。Taking into account what a user is paying attention to, you can make this experience better by offsetting notifications from where the user is currently gazing. 这会限制干扰,并在用户完成读取后自动将其关闭。This limits distractions and automatically dismisses them once the user is finished reading.
  • 留心全息影像: 在 gazed 时,会对影像进行细微反应。Attentive holograms: Holograms that subtly react when being gazed upon. 这可能包括稍微光亮的 UI 元素,这是一种对虚拟狗的缓慢百花齐放花,开始回顾用户并 wagging 其尾部。This can range from slightly glowing UI elements, a slowly blooming flower to a virtual dog starting to look back at the user and wagging its tail. 这种交互可能会在应用程序中提供更有趣的连接和满意度。This interaction might provide an interesting sense of connectivity and satisfaction in your application.

注意力跟踪Attention tracking

有关用户所在位置或位置的信息可以是非常强大的工具。Information on where or what users look at can be an immensely powerful tool. 它可帮助评估设计的可用性并识别工作流中的问题,使其更高效。It can help assess usability of designs and identify problems in workflows to make them more efficient. 目视跟踪可视化和分析是各种应用程序领域的常见做法。Eye tracking visualization and analytics are a common practice in various application areas. 在 HoloLens 2 中,我们提供了一种新的维度来理解,因为3D 全息图可以放置在真实的上下文中并进行相应的评估。With HoloLens 2, we provide a new dimension to this understanding as 3D holograms can be placed in real-world contexts and assessed accordingly. 混合现实工具包提供了用于记录和加载目视跟踪数据的基本示例,以及如何对其进行可视化。The Mixed Reality Toolkit provides basic examples for logging and loading eye tracking data and how to visualize them. Microsoft 致力于推动创新,同时确保用户对其眼睛跟踪信息的使用方式有了经验和透明性。Microsoft is dedicated to facilitating innovation while ensuring that users have an informed and transparent experience with how their eye tracking information is used. 我们将与我们的开发人员和 UX 团队合作,为第三方提供指导,以确保在用户的中心获得经验。We'll work with our developers and UX teams to provide guidance for third parties to ensure that experiences are centered around the user.

此领域的其他应用场景包括:Other applications in this area may include:

  • 远程目视视觉视觉对象: 远程目视目视可视化:直观显示远程协作者正在查看的内容,以便能够提供即时反馈并加速更准确的信息处理。Remote eye-gaze visualization: Remote eye-gaze visualizations: Visualize what remote collaborators are looking at, to be able to provide immediate feedback and facilitate more accurate information processing.
  • 用户研究研究: 关注跟踪可帮助研究人员更深入地了解用户如何感觉和与自然环境进行交流,而不干扰,设计 instinctual 的人机交互。User research studies: Attention tracking can help researchers get more insights into how users perceive and engage with the natural environment, without interfering, to design more instinctual human-computer-interactions. 目视跟踪可以提供调查中的参与者不能直接表述的信息,这种情况可能很容易被研究人员错过。Eye tracking can provide information that is not directly articulated by participants in the study, which otherwise might be easily missed by the researcher.
  • 培训和性能监视: 通过在执行流中更有效地识别瓶颈,来实践和优化任务的执行。Training and performance monitoring: Practice and optimize the execution of tasks by identifying bottlenecks more effectively in the execution flow. 目视跟踪可以提供自然、实时和客观的信息,以帮助提高工作场所的培训、工作效率和安全性。Eye tracking can provide natural, real-time, and objective information to help improve training, productivity, and safety in the workplace.
  • 设计评估、营销和消费者研究: 使用目视跟踪,商业公司可以在真实环境中执行市场营销和消费者研究,或者分析捕获用户对改善产品或空间设计的关注的内容。Design evaluations, marketing, and consumer research: Eye tracking enables commercial companies to perform marketing and consumer studies in real-world environments or analyze what captures a user’s attention to improve product or space design.

其他用例Other use cases

  • 游戏: 您是否曾经想过强大?Gaming: Ever wanted to have superpowers? 机会来了!Here's your chance! 可以通过起始在 levitate 全息影像。You can levitate holograms by staring at them. 从眼睛开始拍摄激光光标,试用RoboRaidShoot laser beams from your eyes - try it out in RoboRaid for HoloLens 2. 将敌人变成石子,或将其冻结。Turn enemies into stone or freeze them. 使用 X 光透视来扫描建筑物。Use your x-ray vision to explore buildings. 没有做不到,只有想不到!Your imagination is the limit! 请注意,用户要了解详细信息,请注意我们的 基于眼睛的输入设计准则Beware of not overwhelming the user though - to find out more, check out our eye-gaze-based input design guidelines.

  • 表现力头像: 目视跟踪通过使用实时眼睛跟踪数据来使真实的头像显示用户正在查看的内容,以更有意义的3D 头像为目标。Expressive avatars: Eye tracking aids in more expressive 3D avatars by using live eye tracking data to animate the avatar's eyes that indicate what the user is looking at.

  • 文本条目: 目视跟踪可用作低工作量文本输入的替代方案,特别是当语音或手不方便使用时。Text entry: Eye tracking can be used as an alternative for low-effort text entry, especially when speech or hands are inconvenient to use.


使用目视查看交互Using eye-gaze for interaction

构建充分利用快速移动目视目标的交互可能会很困难。Building an interaction that takes advantage of fast-moving eye targeting can be challenging. 一方面,眼睛的移动速度非常快,你需要小心地使用眼睛眼睛,因为否则用户可能会发现体验太多或分散注意力。On the one hand, the eyes move so fast that you need to be careful on how to use eye-gaze input, because otherwise users may find the experience overwhelming or distracting. 另一方面,你还可以创建真正的神奇体验,将激发你的用户!On the other hand, you can also create truly magical experiences that will excite your users! 若要帮助你了解关键优势、挑战和设计建议,了解如何进行 交互To help you, check out our overview of key advantages, challenges, and design recommendations for eye-gaze for interaction.

目视跟踪不可用时的回退解决方案Fallback solutions when eye tracking isn't available

在极少数情况下,目视跟踪数据可能不可用。In rare cases, eye tracking data might not be available. 这可能是由于下面列出了最常见的原因:This can be because of different reasons from which the most common are listed below:

  • 系统未能 校准用户The system failed to calibrate the user.
  • 用户跳过了 校准The user skipped the calibration.
  • 用户进行了校准,但决定不向应用程序授予使用其眼睛跟踪数据的权限。The user is calibrated, but decided to not give permission to your app to use their eye tracking data.
  • 用户具有唯一的眼镜或系统尚不支持的一些目视的情况。The user has unique eyeglasses or some eye condition that the system doesn't yet support.
  • 外部因素抑制了一种可靠的眼睛跟踪,如在眼睛前面的头发的情况下,在 HoloLens 面板上出现污迹、眼镜、强烈直接阳光和 occlusions。External factors inhibiting reliable eye tracking such as smudges on the HoloLens visor or eyeglasses, intense direct sunlight, and occlusions because of hair in front of the eyes.

开发人员应确保为这些用户提供适当的后备支持。Developers should ensure that there's appropriate fallback support for these users. 在 " DirectX 中的目视跟踪 " 页上,我们介绍了检测目视跟踪数据是否可用所需的 api。On the Eye Tracking in DirectX page, we explain the APIs required to detect whether eye tracking data is available.

尽管某些用户可能已特意决定撤消、访问其眼睛跟踪数据,并且可以在不提供对其眼睛跟踪数据的访问权限的情况下权衡用户体验令人满意的体验,但在某些情况下,这可能是不可能的。While some users may have consciously decided to revoke, access to their eye tracking data and are ok with the trade-off of an inferior user experience to the privacy of not providing access to their eye tracking data, in some cases this may be unintentional. 如果你的应用程序使用目视跟踪,并且这是体验的重要部分,我们建议你清楚地将此信息传递给用户。If your app uses eye tracking, and this is an important part of the experience, we recommend clearly communicating this to the user.

请通知用户,目视跟踪对于您的应用程序而言是至关重要的, (甚至可能列出一些增强的功能) 若要充分利用您的应用程序,可以帮助用户更好地了解他们所放弃的内容。Kindly informing the user why eye tracking is critical for your application (maybe even listing some enhanced features) to experience the full potential of your application, can help the user to better understand what they're giving up.
根据上述检查) ,帮助用户确定目视跟踪可能不工作 (的原因,并提供一些建议来快速解决潜在问题。Help the user identify why eye tracking may not be working (based on the above checks) and offer some suggestions to quickly troubleshoot potential issues.

例如,如果您可以检测到系统支持目视跟踪,则用户会进行校准,甚至会获得其权限,而不会收到任何眼睛跟踪数据,这可能会导致某些其他问题,如污迹或眼睛封闭像素。For example, if you can detect that the system supports eye tracking, the user is calibrated and has even given their permission, yet no eye tracking data is received, then this may point to some other issues such as smudges or the eyes being occluded.

对于目视跟踪可能不起作用的用户来说,这种情况很罕见。There are rare cases of users for whom eye tracking may not work.
因此,在应用中启用眼睛跟踪时,可以通过允许消除甚至禁用提醒来过于这种情况。Hence, please be respectful of that by allowing to dismiss or even disable reminders for enabling eye tracking in your app.

使用红眼作为主要输入指针回退应用Fall back for apps using eye-gaze as a primary input pointer

如果你的应用程序使用目视看眼作为指针输入来快速选择场景中的全息影像,但眼睛跟踪数据不可用,则建议回退到打印头并开始显示眼睛良好的光标。If your app uses eye-gaze as a pointer input to quickly select holograms across the scene, yet eye tracking data is unavailable, we recommend falling back to head-gaze and start showing the head-gaze cursor. 建议使用超时 (例如,500– 1500 ms) 来确定是否要切换。We recommend using a timeout (for example, 500–1500 ms) to determine whether to switch or not. 此操作可防止在系统每次由于快速的目视运动、传情动漫或传情动漫而发生短暂跟踪丢失时,游标都出现。This action prevents cursors from appearing every time the system may briefly lose tracking because of fast eye motions or winks and blinks. 如果你是一名 Unity 开发人员,则已在混合现实工具包中处理自动回退到打印头。If you're a Unity developer, the automatic fallback to head-gaze is already handled in the Mixed Reality Toolkit. 如果你是 DirectX 开发人员,则需要自行处理此开关。If you're a DirectX developer, you need to handle this switch yourself.

回退其他特定于跟踪的应用程序Fall back for other eye-tracking-specific applications

您的应用程序可能会以独特的方式使用视觉眼睛。Your app may use eye-gaze in a unique way that is tailored specifically to the eyes. 例如,对头像的眼睛进行动画处理,或者为了热图,依赖于有关视觉对象的准确信息。For example, animating an avatar’s eyes or for eye-based attention heatmaps relying on precise information about visual attention. 在这种情况下,没有任何明确的回退。In this case, there's no clear fallback. 如果目视跟踪不可用,可能需要禁用这些功能。If eye tracking isn't available, these capabilities may need to be disabled. 同样,我们建议你清楚地将其传达给可能未意识到该功能无法正常工作的用户。Again, we recommend to clearly communicate this to the user who may be unaware that the capability isn't working.


在此页面中,你可以获得一个良好的概述,使你开始了解 HoloLens 2 的眼睛跟踪和目视眼睛输入的角色。This page has hopefully provided you with a good overview to get you started understanding the role of eye tracking and eye-gaze input for HoloLens 2. 若要开始开发,请查看我们的信息,了解如何 与全息影像交互在 Unity 中目视 看,以及如何 在 DirectX 中观看眼To get started developing, check out our information on the role of eye-gaze for interacting with holograms, eye-gaze in Unity and eye-gaze in DirectX.

另请参阅See also