MRTK 101: How to use Mixed Reality Toolkit Unity for common spatial interactions

MRTK

Learn about how to use MRTK to achieve some of the most widely used common interaction patterns in mixed reality.

  • How to simulate input interactions in Unity editor?
  • How to grab and move an object?
  • How to resize an object?
  • How to move or rotate an object with precision?
  • How to make an object respond to input events?
  • How to add visual feedback?
  • How to add audio feedback?
  • How to use HoloLens 2 style button prefabs?
  • How to make an object follow you?
  • How to make an object face you?

Note

This article has been updated to reflect the changes in MRTK v2.5.1 release

All contents in this page can be tested in Unity editor with MRTK's Input Simulation. If you haven't, follow MRTK Installation Guide (GitHub) to install the latest version of MRTK.

How to simulate input interactions in Unity editor?

MRTK supports in-editor input simulation. Run your scene by clicking Unity's play button, then use the following keys to simulate input:

  • Press W, A, S, D keys to move the camera.
  • Hold the right mouse button and move the mouse to look around.
  • Press Space bar(Right hand) or left Shift key(Left hand) to bring up the simulated hands
  • Press T or Y keys to keep simulated hands in view
  • Press Q or E(horizontal) / R or F(vertical) to rotate simulated hands

You can learn more about Input Simulation in the MRTK documentation.

How to grab and move an object?

Attach the ObjectManipulator.cs and NearInteractionGrabbable.cs scripts to make an object grabbable. ObjectManipulator supports both near and far interactions. You can grab and move an object with HoloLens 2's articulated hand tracking input(near), hand ray(far), motion controller's beam(far), and HoloLens gaze cursor and air-tap(far).


NearInteractionGrabbable and ObjectManipulator.cs assigned to an object


NearInteractionGrabbable and ObjectManipulator.cs assigned to an object for grab and move

How to resize an object?

ObjectManipulator.cs supports two-handed scale/rotation. The script works with various input types, such as HoloLens 2's articulated hand input, HoloLens 1's gaze + gesture input, and Windows Mixed Reality immersive headset's motion controller input.


NearInteractionGrabbable and ObjectManipulator.cs assigned to an object for manipulation

How to move or rotate an object with precision?

Assign BoundsControl.cs to an object to use Bounding Box, which is the interface for scaling and rotating an object. By default, it shows HoloLens 1 style blue handles and wires. To use HoloLens 2 style proximity-based animated handles, you need to assign prefabs and materials.


BoundsControl.cs assigned to an object image


BoundsControl.cs assigned to an object gif

How to make an object respond to input events?

Assign PointerHandler.cs to an object. In the inspector, you can use events OnPointerDown(), OnPointerUp(), OnPointerClicked(), OnPointerDragged() To use these events in a script, implement IMixedRealityPointerHandler.


PointerHandler.cs assigned to an object image

How to add visual feedback?

Assign Interactable.cs to an object. In the inspector, add target object and create a new theme. Using Interactable's theme profiles, you can easily add visual feedback to all available input interaction states.


Image of PointerHandler.cs assigned to an object


Interactable gif

Interactable provides various types of themes including the shader theme, which allows you to control properties of the shader per interaction state.

Another important building block for visual feedback is the MRTK Standard Shader. With MRTK Standard Shader, you can easily add visual feedback effects such as hover light and proximity light. Since MRTK Standard shader performs less computation than the Unity Standard shader, you can create a performant experience.

Create a new material and select the Shader 'Mixed Reality Toolkit > Standard'. Or you can pick one of the existing materials that use MRTK Standard Shader.


MRTK Standard Shader image 1

MRTK Standard Shader image 2

MRTK Standard Shader image 3

How to add audio feedback?

Add AudioSource to an object. Then, in the scripts that expose input events(e.g. Interactable.cs or PointerHandler.cs), assign the object with AudioSource to the event and select AudioSource.PlayOneShot(). You can use your audio clips or choose one from MRTK's audio assets.


Audio Source assigned to an object. AudioSource.PlayOneShot configured in the Interactable's OnPress() and OnRelease() events.

How to use HoloLens 2 style button prefabs?

MRTK provides various types of HoloLens 2's shell (OS) style buttons, including visual feedback like proximity light, compressing box, and a ripple effect on the button surface that improve the user's confidence.


Interactable button

Drag and drop one of the HoloLens 2 style pressable button prefab into your scene. The prefab uses Interactable.cs introduced above. You can use exposed events such as OnClick() in the Interactable to trigger actions.


HoloLens 2 Button Prefab

How to make an object follow you?

Assign RadialView.cs or Follow.cs script to an object. It's part of the Solver script series that allows you to achieve various types of object positioning in 3D space. SolverHandler.cs will be automatically added. Below is an example of RadialView configuration to achieve 'lazy follow' tag-along behavior just like the Start menu in the HoloLens shell. You can specify the minimum/maximum distance and minimum/maximum view degrees. The example below shows positioning the object between 0.4 m and 0.8-m range within 15°. Adjust Lerp Time values to make the positional update faster or slower.


MRTK Standard Shader for solver


Interactable radial solver

How to make an object face you?

Assign Billboard.cs script to an object. It will always face you, whatever your position. You can specify the pivot axis option.


Image of Billboard.cs script assigned to an object with Pivot Axis option Y


Billboard.cs script assigned to an object with Pivot Axis option Y

Ready to create amazing experiences for mixed reality? Visit the pages below and learn more about MRTK and mixed reality.

About the author

Picture of Dong Yoon Park Dong Yoon Park
UX Designer @Microsoft

See also