HoloLens 2 的眼球追蹤Eye tracking on HoloLens 2

MRTK 中的眼睛追蹤示範

HoloLens 2 讓開發人員能夠使用使用者所查看的資訊,讓開發人員能夠使用新的內容層級,並在全像人類的經驗中理解。HoloLens 2 allows for a new level of context and human understanding within the holographic experience by providing developers with the ability to use information about what the user is looking at. 本頁面說明開發人員如何從各種使用案例的眼睛追蹤中獲益,以及設計眼睛型使用者互動時要尋找的內容。This page explains how developers can benefit from eye tracking for various use cases, and what to look for when designing eye-gaze-based user interactions.

眼睛追蹤 API 已設計為使用者的隱私權,避免傳遞任何可識別的資訊,特別是任何生物特徵辨識。Eye tracking API has been designed with a user’s privacy in mind, avoiding passing any identifiable information, particularly any biometrics. 針對感知感知的應用程式,使用者必須授與應用程式許可權以使用眼睛追蹤資訊。For eye-tracking capable applications, the user needs to grant app permission to use eye tracking information.

裝置支援Device support

功能Feature HoloLens (第 1 代)HoloLens (1st gen) HoloLens 2HoloLens 2 沉浸式頭戴裝置Immersive headsets
眼睛Eye-gaze ✔️✔️

校正Calibration

為了讓眼睛追蹤能準確地運作,每位使用者都必須經過 眼睛追蹤使用者校正 ,讓使用者能夠查看一組全像全像一組的目標。For eye tracking to work accurately, each user is required to go through an eye tracking user calibration for which the user has to look at a set of holographic targets. 這可讓裝置為使用者調整系統,以獲得更舒適且更高品質的觀賞體驗,並同時確保一致的眼睛追蹤。This allows the device to adjust the system for a more comfortable and higher quality viewing experience for the user and to ensure accurate eye tracking at the same time.

目視追蹤應該適用于大部分的使用者,但在少數情況下,使用者無法成功校正。Eye tracking should work for most users, but there are rare cases in which a user can't calibrate successfully. 校正可能會因各種原因而失敗,包括但不限於:Calibration might fail for various reasons, including but not limited to:

  • 使用者先前退出宣告校正流程The user previously opted out of the calibration process
  • 使用者有注意力,未遵循校正目標The user got distracted and didn't follow the calibration targets
  • 使用者有特定類型的 contact 鏡頭和眼鏡,系統尚不支援The user has certain types of contact lenses and glasses, which the system doesn't yet support
  • 使用者有特定的眼睛生理學、眼睛狀況,或有系統尚未支援的眼睛外科The user has certain eye physiology, eye conditions or had eye surgery, which the system doesn't yet support
  • 外部因素抑制可靠的眼睛追蹤,例如 HoloLens 面板上的汙跡或眼鏡、密集的直接陽光和遮蔽,因為眼睛正面的頭髮External factors inhibiting reliable eye tracking such as smudges on the HoloLens visor or eyeglasses, intense direct sunlight, and occlusions due to hair in front of the eyes

開發人員應務必為使用者提供適當的支援,讓他們能夠在無法順利校正) (的情況之下,使用眼睛追蹤資料。Developers should make sure to provide adequate support for users for whom eye tracking data may not be available (who aren't able to calibrate successfully). 我們已在此頁面底部的區段中,提供回復解決方案的建議。We have provided recommendations for fallback solutions in the section at the bottom of this page.

若要深入瞭解校正,以及如何確保順暢的體驗,請查看我們的 眼睛追蹤使用者校正 頁面。To learn more about the calibration and about how to ensure a smooth experience, check our eye tracking user calibration page.


可用的眼睛追蹤資料Available eye tracking data

在深入探討眼睛輸入的特定使用案例之前,我們想要簡短指出 HoloLens 2 的 眼睛追蹤 API 所提供的功能。Before going into detail about specific use cases for eye-gaze input, we want to briefly point out the capabilities that the HoloLens 2 Eye Tracking API provides. 開發人員可存取單一眼睛的光線 (注視的原點和方向) 大約 30 FPS (30 Hz)Developers get access to a single eye-gaze ray (gaze origin and direction) at approximately 30 FPS (30 Hz). 如需有關如何存取眼睛追蹤資料的詳細資訊,請參閱我們的開發人員指南,以瞭解如何在您的 DirectXUnity 中使用眼睛。For more detailed information about how to access eye tracking data, refer to our developer guides for using eye-gaze in DirectX and eye-gaze in Unity.

預測的眼睛大約是在實際目標 (的視覺角度1.5 度以內,請參閱下圖) 。The predicted eye-gaze is approximately within 1.5 degrees in visual angle around the actual target (see the illustration below). 由於預期會有些許的 imprecisions,因此開發人員應該規劃這項下限值周圍的一些邊界 (例如,2.0-3.0 度可能會導致) 更熟悉的體驗。As slight imprecisions are expected, developers should plan for some margin around this lower-bound value (for example, 2.0-3.0 degrees may result in a much more comfortable experience). 我們將在下方討論如何處理小型目標的選取。We'll discuss how to address the selection of small targets in more detail below. 為了讓眼球追蹤精準運作,每個使用者都必須接受眼球追蹤使用者校正。For eye tracking to work accurately, each user is required to go through an eye tracking user calibration.

距離 2 公尺的最佳目標大小Optimal target size at 2 meter distance
雙計量距離的最佳目標大小Optimal target size at a 2-meter distance


使用案例Use cases

眼球追蹤可讓應用程式即時追蹤使用者的視線方向。Eye tracking enables applications to track where the user is looking in real time. 下列使用案例說明在混合現實 HoloLens 2 上進行眼追蹤時可能發生的一些互動。The following use cases describe some interactions that are possible with eye tracking on HoloLens 2 in mixed reality. 這些使用案例還不是全像 Shell 體驗的一部分 (也就是當您啟動 HoloLens 2) 時所看到的介面。These use cases aren't yet part of the Holographic Shell experience (that is, the interface that you see when you start up your HoloLens 2). 您可以在「 混合現實」工具組中試用其中一部分,這會提供數個有趣且功能強大的範例來使用眼睛追蹤,例如快速且輕鬆地顯示支援的目標選取專案,並根據使用者的外觀自動滾動文字。You can try some of them in the Mixed Reality Toolkit, which provides several interesting and powerful examples for using eye tracking, such as quick and effortless eye-supported target selections, and automatically scrolling through text based on what the user looks at.

使用者意圖User intent

使用者查看位置和內容的相關資訊,可 為其他輸入 提供功能強大的內容,例如語音、手和控制器。Information about where and what a user looks at provides a powerful context for other inputs, such as voice, hands, and controllers. 這可以運用在各種不同的工作上。This can be used for various tasks. 比方說,這可以透過查看一組影像並說「選取」 (也可以快速且輕鬆地以場景為 目標,也就是查看 和認可) 或「 put this ...」,然後查看使用者想要放置全像 For example, this can range from quickly and effortlessly targeting across the scene by looking at a hologram and saying "select" (also see gaze and commit) or "put this...", then looking over to where the user wants to place the hologram and say "...there". 您可以在混合實境工具組 - 視線導向目標選取混合實境工具組 - 視線導向目標定位中找到相關範例。Examples for this can be found in Mixed Reality Toolkit - Eye-supported Target Selection and Mixed Reality Toolkit - Eye-supported Target Positioning.

此外,使用者意圖的範例可能包括使用使用者查看的相關資訊,以增強與各有虛擬專員和互動式全息的互動。Additionally, an example for user intent might include using information about what users look at to enhance engagement with embodied virtual agents and interactive holograms. 例如,虛擬專員可能會根據目前已查看的內容,來調整可用的選項和其行為。For instance, virtual agents might adapt available options and their behavior, based on currently viewed content.

隱含動作Implicit actions

隱含動作的類別與使用者意圖有密切的關係。The category of implicit actions closely relates to user intent. 它的概念是,全像是全像是 instinctual 的影像或使用者介面專案,可能不會覺得使用者完全與系統互動,而是由系統和使用者保持同步。其中一個範例是以 眼睛為基礎的自動滾動 ,其中使用者可以讀取長文字,這會在使用者到達文字方塊底部時自動開始滾動,以將使用者保持在閱讀流程中,而不需要抬起手指。The idea is that holograms or user interface elements react in an instinctual way that may not even feel like the user is interacting with the system at all, but rather that the system and the user are in sync. One example is eye-gaze-based auto scroll where the user can read a long text, which automatically starts scrolling once the user gets to the bottom of the textbox to keep the user in the flow of reading, without lifting a finger.
其中一個重要層面是,捲動速度會調整為使用者的閱讀速度。A key aspect of this is that the scrolling speed adapts to the reading speed of the user. 另一個範例是 眼睛支援的縮放和平移 ,讓使用者可以覺得完全朝著他或她的焦點。Another example is eye-supported zoom and pan where the user can feel like diving exactly toward what he or she's focused on. 觸發和控制縮放速度可透過聲音或手寫輸入來控制,這對於提供使用者感覺來掌控控制項,同時避免發生大量的情況相當重要。Triggering and controlling zoom speed can be controlled by voice or hand input, which is important for providing the user with the feeling of control while avoiding being overwhelmed. 我們將在下面更詳細討論這些設計考慮。We'll talk about these design considerations in more detail below. 放大之後,使用者可以順暢地遵循,例如,街道的某一堂,使用眼睛的眼睛來探索其鄰近地區。Once zoomed in, the user can smoothly follow, for example, the course of a street to explore his or her neighborhood by using their eye-gaze. 您可以在混合實境工具組 - 視線導向瀏覽範例中找到這類互動的示範。Demo examples for these types of interactions can be found in the Mixed Reality Toolkit - Eye-supported Navigation sample.

其他 隱含動作 的使用案例可能包括:Other use cases for implicit actions may include:

  • 智慧型通知: 在您想要的位置上快顯通知的感到苦惱?Smart notifications: Ever get annoyed by notifications popping up right where you're looking? 考慮到使用者所注意的內容,您可以藉由從使用者目前撥雲見日的位置來抵銷通知,讓這項體驗更好。Taking into account what a user is paying attention to, you can make this experience better by offsetting notifications from where the user is currently gazing. 這會限制分散注意力,並在使用者完成閱讀之後自動將其關閉。This limits distractions and automatically dismisses them once the user is finished reading.
  • 用心全像影像: Gazed 時,會稍微反應的全像投影。Attentive holograms: Holograms that subtly react when being gazed upon. 這可以從稍微照亮的 UI 元素範圍內,以緩慢的綻放花卉為虛擬狗,從使用者回頭看看,並 wagging 它的結尾。This can range from slightly glowing UI elements, a slowly blooming flower to a virtual dog starting to look back at the user and wagging its tail. 這項互動可能會在您的應用程式中提供連線能力和滿意度的有趣意義。This interaction might provide an interesting sense of connectivity and satisfaction in your application.

注意力追蹤Attention tracking

使用者所查看之位置或位置的資訊,是一種功能強大的工具。Information on where or what users look at can be an immensely powerful tool. 它有助於評估設計的可用性,並識別工作流程中的問題,使其更有效率。It can help assess usability of designs and identify problems in workflows to make them more efficient. 目視追蹤視覺效果和分析是各種應用程式區域中常見的作法。Eye tracking visualization and analytics are a common practice in various application areas. 有了 HoloLens 2,我們會提供新的維度來瞭解這一點,因為3D 全息圖可放置在真實世界的內容中,並據以進行評估。With HoloLens 2, we provide a new dimension to this understanding as 3D holograms can be placed in real-world contexts and assessed accordingly. Mixed Reality 工具組提供記錄和載入眼睛追蹤資料的基本範例,以及如何將其視覺化。The Mixed Reality Toolkit provides basic examples for logging and loading eye tracking data and how to visualize them. Microsoft 致力於促進創新,同時確保使用者對其眼睛追蹤資訊的使用方式有明智且透明的體驗。Microsoft is dedicated to facilitating innovation while ensuring that users have an informed and transparent experience with how their eye tracking information is used. 我們將與我們的開發人員和 UX 小組合作,為協力廠商提供指導方針,以確保體驗是以使用者為中心。We'll work with our developers and UX teams to provide guidance for third parties to ensure that experiences are centered around the user.

同屬此領域的其他應用程式包括:Other applications in this area may include:

  • 遠端眼睛視覺效果: 遠端眼睛視覺效果:視覺化遠端共同作業者所關注的內容,以提供立即的意見反應,並協助更精確的資訊處理。Remote eye-gaze visualization: Remote eye-gaze visualizations: Visualize what remote collaborators are looking at, to be able to provide immediate feedback and facilitate more accurate information processing.
  • 使用者研究研究: 注意追蹤可協助研究人員深入瞭解使用者如何觀察並與自然環境互動,而不會干擾如何設計更 instinctual 的人類電腦互動。User research studies: Attention tracking can help researchers get more insights into how users perceive and engage with the natural environment, without interfering, to design more instinctual human-computer-interactions. 目視追蹤可以提供在研究中的參與者未直接表達的資訊,但研究員可能很容易錯過。Eye tracking can provide information that is not directly articulated by participants in the study, which otherwise might be easily missed by the researcher.
  • 訓練和效能監視: 藉由在執行流程中更有效地找出瓶頸,來練習並將工作的執行優化。Training and performance monitoring: Practice and optimize the execution of tasks by identifying bottlenecks more effectively in the execution flow. 目視追蹤可提供自然、即時和目標的資訊,以協助改善工作場所的訓練、生產力和安全性。Eye tracking can provide natural, real-time, and objective information to help improve training, productivity, and safety in the workplace.
  • 設計評估、行銷和取用者研究: 目視追蹤可讓商業公司在真實世界的環境中執行行銷和取用者研究,或分析哪些東西會讓使用者注意,以改善產品或空間的設計。Design evaluations, marketing, and consumer research: Eye tracking enables commercial companies to perform marketing and consumer studies in real-world environments or analyze what captures a user’s attention to improve product or space design.

其他使用案例Other use cases

  • 遊戲: 希望有超能力嗎?Gaming: Ever wanted to have superpowers? 機會來了!Here's your chance! 您可以透過開始來 levitate 全像投影。You can levitate holograms by staring at them. 從您的眼睛放大鐳射字形狀-在 RoboRaid 中試用以 進行 HoloLens 2Shoot laser beams from your eyes - try it out in RoboRaid for HoloLens 2. 將敵人變成石頭或凍結它們。Turn enemies into stone or freeze them. 您可以用 X 光視線掃描建築物。Use your x-ray vision to explore buildings. 您想得到的都行!Your imagination is the limit! 要注意的是,不讓使用者覺得更多,請查看我們的 眼睛型輸入設計指導方針Beware of not overwhelming the user though - to find out more, check out our eye-gaze-based input design guidelines.

  • 表達虛擬人偶: 眼睛追蹤可協助更具表達性的3D 虛擬人偶,方法是使用即時眼睛追蹤資料,以動畫顯示使用者所查看的內容。Expressive avatars: Eye tracking aids in more expressive 3D avatars by using live eye tracking data to animate the avatar's eyes that indicate what the user is looking at.

  • 文字輸入: 您可以使用眼睛追蹤作為低工作量文字輸入的替代方案,特別是當語音或手手不方便使用時。Text entry: Eye tracking can be used as an alternative for low-effort text entry, especially when speech or hands are inconvenient to use.


使用眼睛進行互動Using eye-gaze for interaction

建立利用快速移動眼睛目標的互動可能是一項挑戰。Building an interaction that takes advantage of fast-moving eye targeting can be challenging. 一方面,眼睛的移動速度很快,您需要特別注意如何使用眼睛的輸入,因為使用者可能會發現經驗太過或混亂。On the one hand, the eyes move so fast that you need to be careful on how to use eye-gaze input, because otherwise users may find the experience overwhelming or distracting. 另一方面,您也可以建立真正的神奇體驗,將激發您的使用者!On the other hand, you can also create truly magical experiences that will excite your users! 為協助您,請查看我們的主要優點、挑戰和設計建議,以進行 互動To help you, check out our overview of key advantages, challenges, and design recommendations for eye-gaze for interaction.

當眼睛追蹤無法使用時的回溯解決方案Fallback solutions when eye tracking isn't available

在罕見的情況下,可能無法使用眼睛追蹤資料。In rare cases, eye tracking data might not be available. 這可能是因為下列各項最常見的原因:This can be because of different reasons from which the most common are listed below:

  • 系統無法 校正使用者The system failed to calibrate the user.
  • 使用者略過 校正The user skipped the calibration.
  • 已校正使用者,但決定不授與應用程式使用其眼睛追蹤資料的許可權。The user is calibrated, but decided to not give permission to your app to use their eye tracking data.
  • 使用者有唯一的眼鏡或某些系統尚未支援的眼睛狀況。The user has unique eyeglasses or some eye condition that the system doesn't yet support.
  • 外部因素抑制可靠的眼睛追蹤,例如 HoloLens 面板上的汙跡或眼鏡、密集的直接陽光和遮蔽,因為眼睛正面的頭髮。External factors inhibiting reliable eye tracking such as smudges on the HoloLens visor or eyeglasses, intense direct sunlight, and occlusions because of hair in front of the eyes.

開發人員應該確保這些使用者有適當的回溯支援。Developers should ensure that there's appropriate fallback support for these users. 在 [ DirectX 中的眼睛追蹤 ] 頁面上,我們會說明偵測是否有眼睛追蹤資料的必要 api。On the Eye Tracking in DirectX page, we explain the APIs required to detect whether eye tracking data is available.

雖然某些使用者可能會基本思考模式決定要撤銷,但在某些情況下,您可能會不小心地存取其眼睛追蹤資料,並可在不提供存取權限的情況下權衡使用者體驗的隱私權。While some users may have consciously decided to revoke, access to their eye tracking data and are ok with the trade-off of an inferior user experience to the privacy of not providing access to their eye tracking data, in some cases this may be unintentional. 如果您的應用程式使用眼睛追蹤,而且這是體驗的重要部分,我們建議您明確地將此資訊傳達給使用者。If your app uses eye tracking, and this is an important part of the experience, we recommend clearly communicating this to the user.

請通知使用者,為什麼眼睛追蹤對您的應用程式而言是很重要的 (甚至可以列出一些增強的功能) ,以充分利用您的應用程式,協助使用者更瞭解他們所放棄的功能。Kindly informing the user why eye tracking is critical for your application (maybe even listing some enhanced features) to experience the full potential of your application, can help the user to better understand what they're giving up.
協助使用者根據上述的檢查) ,找出可能無法 (眼睛追蹤的原因,並提供一些建議來快速排解可能的問題。Help the user identify why eye tracking may not be working (based on the above checks) and offer some suggestions to quickly troubleshoot potential issues.

比方說,如果您可以偵測到系統支援眼睛追蹤,則使用者會經過校正並獲得其許可權,但不會收到任何眼睛追蹤資料,而這可能會指向一些其他問題,例如汙跡或眼睛 pixels occluded。For example, if you can detect that the system supports eye tracking, the user is calibrated and has even given their permission, yet no eye tracking data is received, then this may point to some other issues such as smudges or the eyes being occluded.

在罕見的情況下,眼睛追蹤可能無法運作的使用者。There are rare cases of users for whom eye tracking may not work.
因此,請尊重,允許關閉或甚至停用在應用程式中啟用眼睛追蹤的提醒。Hence, please be respectful of that by allowing to dismiss or even disable reminders for enabling eye tracking in your app.

使用眼睛作為主要輸入指標來切換回應用程式Fall back for apps using eye-gaze as a primary input pointer

如果您的應用程式使用眼睛來做為指標輸入,以快速地選取整個場景的全像投影,但無法使用眼睛追蹤資料,我們建議您回到頁首,然後開始顯示頭部眼的游標。If your app uses eye-gaze as a pointer input to quickly select holograms across the scene, yet eye tracking data is unavailable, we recommend falling back to head-gaze and start showing the head-gaze cursor. 我們建議使用 timeout (例如,500-1500 ms) 來判斷是否要切換。We recommend using a timeout (for example, 500–1500 ms) to determine whether to switch or not. 此動作可防止每次系統因為快速的操作或動畫快遞或閃爍而短暫遺失追蹤時,就會出現資料指標。This action prevents cursors from appearing every time the system may briefly lose tracking because of fast eye motions or winks and blinks. 如果您是 Unity 開發人員,則已在混合現實工具組中處理自動回復至前端。If you're a Unity developer, the automatic fallback to head-gaze is already handled in the Mixed Reality Toolkit. 如果您是 DirectX 開發人員,您必須自行處理此參數。If you're a DirectX developer, you need to handle this switch yourself.

切換回其他眼睛追蹤特定應用程式Fall back for other eye-tracking-specific applications

您的應用程式可能會以專為眼睛量身打造的獨特方式來使用眼睛。Your app may use eye-gaze in a unique way that is tailored specifically to the eyes. 例如,以動畫顯示圖片的眼睛,或是基於眼睛的注意,熱度圖依賴視覺效果的精確資訊。For example, animating an avatar’s eyes or for eye-based attention heatmaps relying on precise information about visual attention. 在此情況下,不會有明確的回復。In this case, there's no clear fallback. 如果無法使用眼睛追蹤,可能需要停用這些功能。If eye tracking isn't available, these capabilities may need to be disabled. 同樣地,我們建議您清楚地與可能不知道功能無法運作的使用者溝通。Again, we recommend to clearly communicate this to the user who may be unaware that the capability isn't working.


此頁面希望您有大致的瞭解,讓您開始瞭解 HoloLens 2 的眼睛追蹤和眼睛輸入的角色。This page has hopefully provided you with a good overview to get you started understanding the role of eye tracking and eye-gaze input for HoloLens 2. 若要開始進行開發,請查看有關眼睛的資訊,以瞭解如何與全像 影像互動在 Unity 中的眼睛 ,以及 DirectX 中的眼睛To get started developing, check out our information on the role of eye-gaze for interacting with holograms, eye-gaze in Unity and eye-gaze in DirectX.

另請參閱See also