What is the Mixed Reality Toolkit

Mixed Reality Toolkit


MRTK-Unity is a Microsoft-driven project that provides a set of components and features, used to accelerate cross-platform MR app development in Unity. Here are some of its functions:

  • Provides the cross-platform input system and building blocks for spatial interactions and UI.
  • Enables rapid prototyping via in-editor simulation that allows you to see changes immediately.
  • Operates as an extensible framework that provides developers the ability to swap out core components.
  • Supports a wide range of platforms:
Platform Supported Devices
OpenXR (Unity 2020.3.8+) Microsoft HoloLens 2
Windows Mixed Reality headsets
Windows Mixed Reality Microsoft HoloLens
Microsoft HoloLens 2
Windows Mixed Reality headsets
Oculus (Unity 2019.3 or newer) Oculus Quest
OpenVR Windows Mixed Reality headsets
HTC Vive
Oculus Rift
Ultraleap Hand Tracking Ultraleap Leap Motion controller
Mobile iOS and Android
Platform Supported Devices
OpenXR (Preview in MRTK 2.6, Unity 2020.3.8+) Microsoft HoloLens 2
Windows Mixed Reality headsets
Windows Mixed Reality Microsoft HoloLens
Microsoft HoloLens 2
Windows Mixed Reality headsets
Oculus (Unity 2019.3 or newer) Oculus Quest
OpenVR Windows Mixed Reality headsets
HTC Vive
Oculus Rift
Ultraleap Hand Tracking Ultraleap Leap Motion controller
Mobile iOS and Android

Note

Please avoid Unity 2020.3.21f and 2020.3.22f, which cause flashing and strobing effect on HoloLens 2. For more information, see here.

Getting started with MRTK

If you're new to MRTK or Mixed Reality development in Unity, we recommend installing and exploring the MRTK Examples Hub sample application on your device or emulator.

Once you've got the hang of what Mixed Reality and MRTK has to offer, install the necessary tools and follow our beginner level HoloLens 2 tutorial series.

Want to see what's going on under the hood?

Documentation

Release notes
Release Notes
MRTK Overview
MRTK Overview
API Reference
API Reference

Build status

Branch CI Status Docs Status
main CI Status Docs Status

Feature areas

UX building blocks

Button Button
A button control which supports various input methods, including HoloLens 2's articulated hand

Bounds Control Bounds Control
Standard UI for manipulating objects in 3D space

Object Manipulator Object Manipulator
Script for manipulating objects with one or two hands

Slate Slate
2D style plane which supports scrolling with articulated hand input

System Keyboard System Keyboard
Example script of using the system keyboard in Unity

Interactable Interactable
A script for making objects interactable with visual states and theme support

Solver Solver
Various object positioning behaviors such as tag-along, body-lock, constant view size and surface magnetism

Object Collection Object Collection
Script for laying out an array of objects in a three-dimensional shape

Tooltip Tooltip
Annotation UI with a flexible anchor/pivot system, which can be used for labeling motion controllers and objects

Slider Slider
Slider UI for adjusting values supporting direct hand tracking interaction

MRTK Standard Shader MRTK Standard Shader
MRTK's Standard shader supports various Fluent design elements with performance

Hand Menu Hand Menu
Hand-locked UI for quick access, using the Hand Constraint Solver

App Bar App Bar
UI for Bounds Control's manual activation

Pointers Pointers
Learn about various types of pointers

Fingertip Visualization Fingertip Visualization
Visual affordance on the fingertip which improves the confidence for the direct interaction

Near Menu Near Menu
Floating menu UI for the near interactions

Spatial Awareness Getting started Spatial Awareness View
Make your holographic objects interact with the physical environments

Voice Command Voice Command
Scripts and examples for integrating speech input

Progress Indicator Progress Indicator
Visual indicator for communicating data process or operation

Dialog Dialog
UI for asking for user's confirmation or acknowledgement

Hand Coach Hand Coach
Component that helps guide the user when the gesture has not been taught

Hand Physics Service Hand Physics Service [Experimental]
The hand physics service enables rigid body collision events and interactions with articulated hands

Scrolling Collection Scrolling Collection
An Object Collection that natively scrolls 3D objects

Dock Dock [Experimental]
The Dock allows objects to be moved in and out of predetermined positions

Eye Tracking: Target Selection Eye Tracking: Target Selection
Combine eyes, voice and hand input to quickly and effortlessly select holograms across your scene

Eye Tracking: Navigation Eye Tracking: Navigation
Learn how to auto-scroll text or fluently zoom into focused content based on what you are looking at

Eye Tracking: Heat Map Eye Tracking: Heat Map
Examples for logging, loading and visualizing what users have been looking at in your app

Tools

Optimize Window Optimize Window Dependency Window Dependency Window Build Window Build Window Input recording Input recording
Automate configuration of Mixed Reality projects for performance optimizations Analyze dependencies between assets and identify unused assets Configure and execute an end-to-end build process for Mixed Reality applications Record and playback head movement and hand tracking data in editor

Example scenes

MRTK provides example scenes that demonstrate how to use MRTK's features. You can find the example scenes under Assets/MRTK/Examples/Demos folder. Read the Example scenes page to learn how to acquire and run example scenes. Hand Interaction Examples scene is a great place to start experiencing MRTK's building blocks for interactions and UI.

Example Scene 2

MRTK examples hub

With the MRTK Examples Hub, you can try various example scenes in MRTK without building and deploying each scene. You can download pre-built app packages for HoloLens(x86), HoloLens 2(ARM), and Windows Mixed Reality immersive headsets(x64) by selecting the "Mixed Reality Toolkit Examples" package in the MR Feature Tool. Make sure to use the Windows Device Portal to install apps on HoloLens (1st gen). On HoloLens 2, you can download and install MRTK Examples Hub through the Microsoft Store app.

See Examples Hub README page to learn about the details on creating a multi-scene hub with MRTK's scene system and scene transition service.

Example Scene Hub

Sample apps made with MRTK

Periodic Table of the Elements Galaxy Explorer Surfaces sample app
Periodic Table of the Elements is an open-source sample app which demonstrates how to use MRTK's input system and building blocks to create an app experience for HoloLens and Immersive headsets. Read the porting story: Bringing the Periodic Table of the Elements app to HoloLens 2 with MRTK v2 Galaxy Explorer is an open-source sample app that was originally developed in March 2016 as part of the HoloLens 'Share Your Idea' campaign. Galaxy Explorer has been updated with new features for HoloLens 2, using MRTK v2. Read the story: The Making of Galaxy Explorer for HoloLens 2 Surfaces is an open-source sample app for HoloLens 2 which explores how we can create a tactile sensation with visual, audio, and fully articulated hand-tracking. Check out Microsoft MR Dev Days session Learnings from the Surfaces app for the detailed design and development story.

Session videos from Mixed Reality Dev Days 2020

MRDevDays 1 MRDevDays 3 MRDevDays 2
Tutorial on how to create a simple MRTK app from start to finish. Learn about interaction concepts and MRTK’s multi-platform capabilities. Deep dive on the MRTK’s UX building blocks that help you build beautiful mixed reality experiences. An introduction to performance tools, both in MRTK and external, as well as an overview of the MRTK Standard Shader.

See Mixed Reality Dev Days to explore more session videos.

Engage with the community

This project has adopted the Microsoft Open Source Code of Conduct. For more information, see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

Useful resources on the Mixed Reality Dev Center

Discover Discover Design Design Develop Develop Distribute) Distribute
Learn to build mixed reality experiences for HoloLens and immersive headsets (VR). Get design guides. Build user interface. Learn interactions and input. Get development guides. Learn the technology. Understand the science. Get your app ready for others and consider creating a 3D launcher.

Useful resources on Azure

Spatial Anchors
Spatial Anchors
Speech Services Speech Services Vision Services Vision Services
Spatial Anchors is a cross-platform service that allows you to create Mixed Reality experiences using objects that persist their location across devices over time. Discover and integrate Azure powered speech capabilities like speech to text, speaker recognition or speech translation into your application. Identify and analyze your image or video content using Vision Services like computer vision, face detection, emotion recognition or video indexer.

How to contribute

Learn how you can contribute to MRTK at Contributing.

Getting help

If you run into issues caused by MRTK or otherwise have questions about how to do something, there are a few resources that can help: