手部直接操作Direct manipulation with hands


直接操作是直接以雙手觸控全像投影的輸入模型。Direct manipulation is an input model that involves touching holograms directly with your hands. 這個概念背後的構想是要讓物件如同在真實環境中作動。The idea behind this concept is that objects behave just as they would in the real world. 按鈕只要按下即可啟用、物件可藉由抓取來選取,2D 內容的行為則類似於虛擬觸控螢幕。Buttons can be activated simply by pressing them, objects can be picked up by grabbing them, and 2D content behaves like a virtual touchscreen. 直接操作以能供性為基礎,這表示它很容易使用。Direct manipulation is affordance-based, meaning it's user-friendly. 使用者不需學習任何象徵性的手勢。There are no symbolic gestures to teach users. 所有互動都是以您可觸摸或抓取的視覺化元素為基礎。All interactions are built around a visual element that you can touch or grab. 我們將直接操作視為「近距離」輸入模型,因為它最適合用於伸手可及範圍內的內容互動。It's considered a "near" input model in that it's best used for interacting with content within arms reach.

裝置支援Device support

輸入模型Input model HoloLens (第 1 代)HoloLens (1st gen) HoloLens 2HoloLens 2 沉浸式頭戴裝置Immersive headsets
手部直接操作Direct manipulation with hands ❌ 未支援❌ Not supported ✔️ 建議使用✔️ Recommended ➕ 支援。➕ Supported. 針對 UI,建議您改為使用手部指向和行動For UI, we recommend point and commit with hands instead.

直接操作是 HoloLens 2 的主要輸入模型,HoloLens 2 使用新式連貫手部追蹤系統。Direct manipulation is a primary input model on HoloLens 2, which uses the new articulated hand-tracking system. 您也可使用運動控制器,搭配沉浸式頭戴裝置來操作輸入模型,但對於物件操作以外的互動,不建議以此作為主要機制。The input model is also available on immersive headsets by using motion controllers, but isn't recommended as a primary means of interaction outside of object manipulation. 在 HoloLens (第 1 代) 上無法進行直接操作。Direct manipulation isn't available on HoloLens (1st gen).

可碰撞的指尖Collidable fingertip

在 HoloLens 2 上,使用者的雙手會被辨識和解譯為左右兩手的架構模型。On HoloLens 2, the user's hands are recognized and interpreted as left and right-hand skeletal models. 若要實作直接以雙手觸控全像投影的概念,在理想情況下,五個 collider 可以連結至左右兩手架構模型的五個指尖。To implement the idea of touching holograms directly with hands, ideally, five colliders could be attached to the five fingertips of each hand skeletal model. 但由於缺少觸覺反饋能力,10 個可碰撞的指尖可能對全像投影造成非預期且無法預測的衝突。However, because of the lack of tactile feedback, 10 collidable fingertips can cause unexpected and unpredictable collisions with holograms.

我們建議僅將 collider 放在兩手的食指上。We suggest only putting a collider on each index finger. 可碰撞的食指指尖仍可作為使用中的觸控點,用於涉及其他手指的各種觸控手勢。The collidable index fingertips can still serve as active touch points for diverse touch gestures involving other fingers. 觸控手勢包括一指點按、一指點選、兩指點按和五指點按,如下所示:Touch gestures include One-finger press, One-finger tap, Two-finger press, and Five-finger press, as shown below:

可碰撞的指尖collidable fingertip
可碰撞的指尖Collidable fingertip

一指點按One-finger press
一指點按One-finger press

一指點選One-finger tap
一指點選One-finger tap

五指點按Five-finger press
五指點按Five-finger press

球體 colliderSphere collider

建議您不要使用隨機的一般形狀,而應使用球體 collider。Instead of using a random generic shape, we suggest you use a sphere collider. 然後,您可以用視覺化的方式加以呈現,為近距離目標鎖定提供明確的提示。Then you can visually render it to provide better cues for near targeting. 球體的直徑應符合食指的粗細,以提高觸控的正確性。The sphere's diameter should match the thickness of the index finger to increase touch accuracy. 呼叫手部 API,更容易擷取手指粗細的變數。It's easier to retrieve the variable of finger thickness by calling the hand API.

指尖游標Fingertip cursor

除了在食指指尖上顯示可碰撞的球體以外,我們也建立了進階的指尖游標,以提供更好的近距離目標鎖定體驗。In addition to rendering a collidable sphere on the index fingertip, we've created an advanced fingertip cursor to achieve a better near-targeting experience. 這是連結至食指指尖的一個環狀游標。It's a donut-shaped cursor attached to the index fingertip. 此游標會根據鄰近性,在方向和大小方面動態回應目標,詳述如下:According to proximity, it dynamically reacts to a target for orientation and size as detailed below:

  • 當食指移向全像投影時,游標一律會與全像投影的表面平行,且其大小會逐漸縮減。When an index finger moves toward a hologram, the cursor is always parallel to the hologram's surface and gradually shrinks its size.
  • 當手指接觸到表面時,游標會縮減為一個點,並發出觸控事件。As soon as the finger touches the surface, the cursor shrinks into a dot and emits a touch event.

透過互動式反饋,使用者可執行高精確度的近距離目標鎖定工作,例如觸發超連結或按下按鈕,如下所示。With interactive feedback, users can achieve high precision near-targeting tasks, such as triggering a hyperlink or pressing a button as shown below.

指尖游標遠距Fingertip cursor far
指尖游標遠距Fingertip cursor far

指尖游標近距Fingertip cursor near
指尖游標近距Fingertip cursor near

指尖游標接觸Fingertip cursor contact
指尖游標接觸Fingertip cursor contact

週框方塊與近接著色器Bounding box with proximity shader

全像投影本身也須具備提供視覺和聽覺反饋的能力,以彌補觸覺反饋能力的不足。The hologram itself also requires the ability to provide both visual and audio feedback to compensate the lack of tactile feedback. 為此,我們設計了週框方塊與近接著色器的概念。For that, we generate the concept of a bounding box with a proximity shader. 週框方塊是足以包覆一個 3D 物件的最小體積區域。A bounding box is a minimum volumetric area that encloses a 3D object. 週框方塊具有互動式呈現機制,名為近接著色器。The bounding box has an interactive rendering mechanism called a proximity shader. 近接著色器的行為如下:The proximity shader behaves:

暫留 (遠距) 視覺化回饋Hover (far) with visual feedback
暫留 (遠距)Hover (far)
當食指進入範圍內時,手指焦點即會投射在週框方塊的表面。When the index finger is within a range, a fingertip spotlight is cast on the surface of the bounding box.

暫留 (近距) 視覺化回饋Hover (near) with visual feedback
暫留 (近距)Hover (near)
當指尖越來越接近表面時,焦點會縮小。When the fingertip gets closer to the surface, the spotlight shrinks.

接觸開始Contact begins
接觸開始Contact begins
手指一碰到表面時,整個週框方塊就會變色,或產生視覺效果以反映觸碰狀態。As soon as the fingertip touches the surface, the entire bounding box changes color or generates visual effects to reflect the touch state.

接觸結束Contact ends
接觸結束Contact ends
此外也可以啟用音效來補強視覺的觸碰反饋。A sound effect can also be activated to enhance the visual touch feedback.

可點按的按鈕Pressable button

有了可碰撞的指尖,使用者即可與基本的全像攝影 UI 元件互動,例如可點按的按鈕。With a collidable fingertip, users are now ready to interact with a fundamental holographic UI component, such as a pressable button. 可點按的按鈕是專為直接手指點按設計的全像攝影按鈕。A pressable button is a holographic button tailored for a direct finger press. 同樣地,由於缺少觸覺反饋能力,可點按的按鈕也設有若干機制,來處理觸覺反饋能力的相關問題。Again, because of the lack of tactile feedback, a pressable button equips a couple mechanisms to tackle tactile feedback-related issues.

  • 第一項機制是具有近接著色器的週框方塊,其詳細說明請見上一節。The first mechanism is a bounding box with a proximity shader, which is detailed in the previous section. 它可讓使用者在接近及觸碰按鈕時更能清楚感知鄰近性。It gives users a better sense of proximity when they approach and make contact with a button.
  • 第二項機制是下壓。The second mechanism is depression. 下壓會在指尖接觸到按鈕後,產生按下的感覺。Depression creates a sense of pressing down after a fingertip contacts a button. 此機制能確保按鈕會緊隨著指尖沿深度軸移動。The mechanism ensures that the button tightly moves with the fingertip along the depth axis. 在按鈕達到所選的深度 (按下) 或離開該深度 (放開) 之後,就會觸發按鈕。The button can be triggered when it reaches a chosen depth (on press) or leaves the depth (on release) after passing through it.
  • 在觸發按鈕時應加上音效以增強反饋。The sound effect should be added to enhance feedback when the button is triggered.

可點按的按鈕遠距pressable button far
手指遠離Finger is far away

可點按的按鈕近距pressable button near
手指靠近Finger approaches

可點按的按鈕接觸開始pressable button contact begins
接觸開始Contact begins

可點按的按鈕點按pressable button press
按下Press down

2D 平板互動2D slate interaction

2D 平板是用來裝載 2D 應用程式內容 (例如網頁瀏覽器) 的全像攝影容器。A 2D slate is a holographic container used to host 2D app content, such as a web browser. 透過直接操作與 2D 平板互動的設計概念,和與實體觸控螢幕互動的設計概念相同。The design concept for interacting with a 2D slate via direct manipulation is the same as interacting with a physical touch screen.

與觸控平板互動To interact with the slate contact

以食指按下超連結或按鈕。Use an index finger to press a hyperlink or a button.

以食指向上和向下捲動平板內容。Use an index finger to scroll a slate content up and down.

使用者可用兩根食指,根據手指的相對移動來放大和縮小平板內容。The user's two index fingers are used to zoom in and out of the slate content, according to the relative motion of the fingers.

操作 2D 平板本身For manipulating the 2D slate itself

顯示抓取和拖曳功能的圖形Graphic showing grab and drag feature
將手靠近角落或邊緣時,可顯示最接近的操作能供性。Move your hands toward corners and edges to reveal the closest manipulation affordances. 抓取 2D 平板上方的 holobar,可以移動整個平板。Grab the Holobar at the top of the 2D slate, which lets you move the whole slate.

顯示縮放功能的圖形Graphic showing scale feature
抓取操作能供性,並透過角落能供性進行統一縮放。Grab the manipulation affordances and do uniform scaling through the corner affordances.

抓取操作能供性,透過邊緣能供性進行自動重排。Grab the manipulation affordances and do reflow via the edge affordances.

3D 物件操作3D object manipulation

HoloLens 2 可讓使用者將週框方塊套用至每個 3D 物件,以便用手指引及操作 3D 全像攝影物件。HoloLens 2 lets users enable their hands to direct and manipulate 3D holographic objects by applying a bounding box to each 3D object. 週框方塊可透過其近接著色器提供更確切的深度認知。The bounding box provides better depth perception through its proximity shader. 週框方塊衍生出兩種 3D 物件操作方法。With the bounding box, there are two design approaches for 3D object manipulation.

能供性操作Affordance-based manipulation

能供性操作可讓您透過週框方塊及其周圍的操作能供性來操作 3D 物件。Affordance-base manipulation lets you manipulate the 3D object through a bounding box along with the manipulation affordances around it.

顯示物件週框方塊和移動功能的圖形Graphic showing an objects bounding box and move feature
當使用者的手接近 3D 物件時,就會顯示週框方塊和最接近的能供性。As soon as a user's hand is close to a 3D object, the bounding box, and the nearest affordance are revealed. 使用者可以抓取週框方塊來移動整個物件。Users can grab the bounding box to move the whole object.

顯示抓取物件邊緣以旋轉的使用者圖形Graphic showing user grabbing an objects edge to rotate
使用者可以抓取邊緣能供性來進行旋轉。Users can grab the edge affordances to rotate.

顯示抓取物件角落以縮放的使用者圖形Graphic showing user grabbing an objects corner to scale
使用者可以抓取角落能供性來執行統一尺寸調整。Users can grab the corner affordances to scale uniformly.

非能供性操作Non-affordance-based manipulation

非能供性操作不會將能供性連結至週框方塊。Non-affordance-based manipulation doesn't attach affordance to the bounding box. 使用者只能顯示週框方塊,然後直接與它互動。Users can only reveal the bounding box, then directly interact with it. 若以一隻手抓住週框方塊,物件的轉譯和旋轉會與那隻手的動作和方向相關聯。If the bounding box is grabbed with one hand, the translation and rotation of the object are associated to motion and orientation of the hand. 以雙手抓住物件時,使用者可以根據兩隻手的相對動作來轉譯、縮放及旋轉物件。When the object is grabbed with two hands, users can translate, scale, and rotate it according to relative motions of two hands.

特定操作需要精確度。Specific manipulation requires precision. 建議您使用 能供性操作,因為它可提供高層級的細微性。We recommend that you use affordance-based manipulation because it provides a high level of granularity. 針對彈性操作,建議您使用 非能供性操作,因為它可帶來即時而有趣的體驗。For flexible manipulation, we recommend you use non-affordance manipulation as it allows for instant and playful experiences.

本能手勢Instinctual gestures

在 HoloLens (第 1 代) 推出時,我們曾教過使用者幾個預先定義的手勢,例如綻開和空中點選。With HoloLens (1st gen), we taught users a couple of predefined gestures, such as bloom and air tap. 而在使用 HoloLens 2 時,使用者並不需要記住任何象徵性的手勢。For HoloLens 2, we don't ask users to memorize any symbolic gestures. 使用者與全像投影和內容互動時所需使用的所有必要手勢,都是符合本能的。All required user gestures, where users need to interact with holograms and content, are instinctual. 協助使用者透過 UI 能供性的設計執行手勢,他們就能學會本能手勢。The way to achieve instinctual gestures is to help users perform gestures through the design of UI affordances.

例如,如果我們引導使用者以兩指捏合抓取某個物件或控制點,表示該物件或控制點應該很小。For example, if we encourage the user to grab an object or a control point with a two finger pinch, the object or the control point should be small. 如果我們要使用者進行五指抓取,表示該物件或控制點應相對較大。If we want the user to do a five finger grab, the object or the control point should be relatively large. 與按鈕類似,微小的按鈕會使得使用者只能用單一手指進行按壓。Similar to buttons, a tiny button would limit users to press it with a single finger. 較大的按鈕會促使使用者使用手掌進行按壓。A large button would encourage users to press it with their palms.

顯示抓取小型物件以移動的使用者圖形Graphic showing user grabbing small object to move
小型物件Small object

顯示抓取中型物件以移動的使用者圖形Graphic showing user grabbing medium object to move
中型物件Medium object

顯示抓取大型物件以移動的使用者圖形Graphic showing user grabbing large object to move
大型物件Large object

手部與 6 DoF 控制器之間的對稱設計Symmetric design between hands and 6 DoF controllers

您可能已經注意到,AR 中的手勢與 VR 中的運動控制器在互動方式上有其相似之處。You may have noticed that there are interaction parallels we can draw between hands in AR and motion controllers in VR. 這兩種輸入在其各自的環境中都可用來觸發直接操作。Both inputs can be used to trigger direct manipulations in their respective environments. 在 HoloLens 2 中,近距離以手抓取並拖曳的效用,非常類似於 WMR 運動控制器上的抓取按鈕所能做到的。In HoloLens 2, grabbing and dragging with hands at a close distance works much the same way as the grab button does on WMR motion controllers. 這種相似性可讓使用者在兩種平台上都能採用熟悉的互動方式,且若您決定在這兩個平台之間移轉應用程式,這一點可能也有幫助。This provides users with interaction familiarity between the two platforms, which might prove useful if you ever decide to port your application between platforms.

透過眼球追蹤達到最佳效果Optimize with eye tracking

直接操作若能如預期運作,將可達到奇妙的效果。Direct manipulation can feel magical if it works as intended. 但若無法順利將手移到想要的位置,而老是無意中觸發全像投影,就會出現挫敗感。But it can also become frustrating if you can’t move your hand anywhere without unintentionally triggering a hologram. 眼球追蹤或許有助於判斷使用者真正的意圖。Eye tracking potentially helps to better identify what the user’s intent is.

  • 時機:避免無意中觸發操作回應。When: Reduce unintentionally triggering a manipulation response. 眼球追蹤有助於了解使用者目前從事的活動。Eye tracking allows for better understanding what a user is currently engaged with. 例如,想像您在閱讀全像攝影 (指示) 文字時,您伸手要去拿實際環境中的工具。For example, imagine you're reading through a holographic (instructional) text when reaching over to grab you real-world work tool.

此時,您的手不小心劃過某個您根本沒留意過的互動式全像攝影按鈕。By doing so, you accidentally move your hand across some interactive holographic buttons that you hadn't even noticed before. 例如,該按鈕可能不在使用者的視野 (FoV) 內。For example, it may be outside the user's field-of-view (FoV).

如果使用者已有一段時間未注視某個全像投影,但仍偵測到觸碰或抓取事件,表示此互動很可能是無意的。If the user hasn't looked at a hologram for a while, yet a touch or grasp event has been detected for it, the interaction is likely unintentional.

  • 標的:除了處理誤判的啟用以外,它還有助於判斷所應抓取或觸碰的全像投影,因為從您的視角可能無法看出精準的交叉點,尤其是有數個全像投影的位置彼此十分靠近時。Which one: Aside from addressing false positive activations, another example includes better identifying which holograms to grab or poke as the precise intersection point may not be clear from your perspective, especially if several holograms are positioned close to each other.

    雖然 HoloLens 2 的眼球追蹤在判斷眼部注視時會因其精準度而有所限制,但對近距離互動而言仍不失為有利的工具,因為透過手部輸入互動時,會有深度的差異。While eye tracking on HoloLens 2 has limitations based on how accurately it can determine your eye gaze, this can still be helpful for near interactions because of depth disparity when interacting with hand input. 舉例而言,要判斷您的手是在全像投影的後面還是前面,以精確地抓取操作工具,有時是很困難的。This means it's sometimes difficult to determine whether your hand is behind or in front of a hologram to precisely grab a manipulation widget, for example.

  • 目的地:運用使用者在執行快速拋投手勢時看往何處的相關資訊。Where to: Use information about what a user is looking at with quick-throwing gestures. 抓取全像投影並約略拋向您要的目的地。Grab a hologram and roughly toss it toward your intended destination.

    雖然這有時行得通,但快速執行手勢可能會導致目的地嚴重失準。While this sometimes works, quickly doing hand gestures may result in highly inaccurate destinations. 不過,同時運用眼球追蹤或可改善手勢的準確度。However, eye tracking could improve the accuracy of the gesture.

MRTK (混合實境工具組) 中適用於 Unity 的操作Manipulation in MRTK (Mixed Reality Toolkit) for Unity

透過 MRTK ,您可以使用指令碼 ObjectManipulator 輕鬆執行常見的操作行為。With MRTK, you can easily achieve common manipulation behavior using the script ObjectManipulator. 使用 ObjectManipulator,您可以直接用手或透過手部射線來抓取和移動物件。With ObjectManipulator, you can grab and move objects directly with hands or with hand ray. 它也支援雙手操作來縮放和旋轉物件。It also supports two-handed manipulation for scaling and rotating an object.

另請參閱See also