Building Immersive Apps Using The Mixed Reality Toolkit

Editor's note: The following post was written by Windows Development MVP  Senthamil Selvan   as part of our Technical Tuesday series. Danae Aguilar  of the MVP Award Blog Technical Committee served as the technical reviewer for this piece.

With the release of the Fall Creator Update (FCU) for Windows 10, Microsoft is entering into the Mixed Reality market. In this article, we’ll delve into how to start programming for Mixed Reality in Windows 10 - but first, make sure you have  all the tools needed to build apps in Windows by following this link.  

You can now download the FCU for the Windows 10 by following the steps here.

Let’s get started!

I strongly recommend looking through the contents of this link, in order to setup the machine and Unity environment to be able to start the programming.

Step 1: Installation

Install the required software for the development machine:

  1. Windows 10 FCU
  2. Unity 3d
  3. Visual Studio 2017
  4. Simulator for MR Devices

Step 2: Download the Mixed Reality ToolKit from the GitHub Download the full toolkit and unzip it to the local machine. In this case, we unzipped it to C:\Mixed Reality. The photo below shows the folder structure after unzipping:  

Step 3: Export Package

Open the Unity application. In the startup screen, select the Open option on top right. Select the folder belonging to the Mixed Reality Toolkit, which was unzipped in previous step. Wait for Unity to load all the Assets.

Once the Unity is loaded, go to the Assets folder from Project tab. In the Asset menu on top, select Export Package. Export it as a MixedReality package to any folder you want to use.

Next, close the Unity and return to the MixedReality export package folder. Now we have the ready made package with all the prefab, scripts and the animation for the Unity to start programming.

Start the Mixed Reality App Programming

Open the Unity application and click on New to create the new project. Make sure you select the 3d option in the New window. Once the Unity is loaded, import the Mixed Reality package, which was exported in step 2.

Next, we’ll go through the following sequence - Assets -> Import Package -> Custom Package.

In the window, select the Mixed Reality package.

Once the package is imported you will now see the Mixed Reality Toolkit in the menu option of the Unity Editor. 

Now select the Mixed Reality Menu. Go to Configure and select Apply Mixed Reality Project Settings

In the Apply Mixed Reality Project Settings check all that applies. Make sure Target Occluded Devices is checked. Click on Apply to apply the settings.

Now select the Mixed Reality Toolkit->Configure-> Apply Mixed Reality Scene Settings

Click on Apply to apply the scene for the Mixed Reality.

In this sample app, we are going to capture the Gaze and the Gesture input to manipulate the 3D object.

The input tap event is handled to throw a ball with the force. The ball is thrown at the sphere or cube which is placed in front of the camera.

Create the 3D objects

1. Create the 3D object Cube and set the below values. Rename the cube to Player1

2. Create the 3D sphere and set the X=-0.3, Z=5.916 and Y=0.51 in the Position. Rename the sphere to Player2

3. Create Plane and apply the below settings

4. Create a Material with your color and apply the material to both cube and ball.

5. In my case, I created a ColorMaterial with color blue, which is applied to Cube and Ball. I applied the colour grey to the PlaneMaterial.

6. Select the Cube, and Add Component on right panel and type Rigidbody. Select the Rigidbody to be added. This will give the gravity to the object, to behave like a real ball or cube.

The final setup will look like the below:

Gaze Input

Now we are going to handle the Gaze input and change the color of Player1 and Player2. When the user inputs Gaze for Player1 (cube), its color will be changed to red.

  • Select the Assets folder from the Project tab in Unity Editor.
  • In the Assets folder on right panel, right click and create C# Script.
  • Name the script GazeAtObject.cs
  • Once the script is created, double click on that to launch Visual Studio. This will open the C# file and make it ready for coding.
  • Add the below code to the cs file
 using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class GazeAtObject : MonoBehaviour {

    private Color original;
    // Use this for initialization
    void Start()
        var cube = this.GetComponent();
        original = cube.material.color;

    // Update is called once per frame
    void Update()


    void OnSelect()
        var cube = this.GetComponent();
        cube.material.color =;
    void OnReset()
        var cube = this.GetComponent();
        cube.material.color = original;
  • Select the Player1 from the hierarchy and from the Asset drag the script GazeAtObject.cs and drop it on the right panel.
  • Select the Player2 from the hierarchy and from the Asset drag the script GazeAtObject.cs and drop it on the right panel.

Basically, the above code will change color when an OnSelect method is triggered, and the color is set back to original OnReset method.

Handling the Input

Now we need to handle the Gaze input and call the OnSelect and OnReset method on it. On the Asset folder search for Cursor.cs. You can also locate the cursor.cs under Assets->HoloToolkit->Input->Scripts->Cursor folder

Now double click to open the cursor.cs file in the visual studio. Locate the OnPointerSpecificFocusChanged and add the below code at the end of the event.

 if (oldFocusedObject != null && == "Player1")

The full function will look like the below:

 protected virtual void OnPointerSpecificFocusChanged(IPointingSource pointer, GameObject oldFocusedObject, GameObject newFocusedObject)
            if (pointer == Pointer)
                TargetedObject = newFocusedObject;

                CursorModifier newModifier = (newFocusedObject == null)
                    ? null
                    : newFocusedObject.GetComponent();

            // my code here
            if (oldFocusedObject != null && == "Player1")

Now locate the UpdateCursorTransform function and add the below code in the first else statement.

 //my code here
                TargetedObject = newTargetedObject;
                if ( == "Player1")

Again, the full function will look like below:

 protected virtual void UpdateCursorTransform()
            FocusDetails focusDetails = FocusManager.Instance.GetFocusDetails(Pointer);
            GameObject newTargetedObject = focusDetails.Object;

            // Get the forward vector looking back along the pointing ray.
            Vector3 lookForward = -Pointer.Ray.direction;

            // Normalize scale on before update
            targetScale =;

            // If no game object is hit, put the cursor at the default distance
            if (newTargetedObject == null)
                TargetedObject = null;
                TargetedCursorModifier = null;
                targetPosition = Pointer.Ray.origin + Pointer.Ray.direction * DefaultCursorDistance;
                targetRotation = lookForward.magnitude > 0 ? Quaternion.LookRotation(lookForward, Vector3.up) : transform.rotation;
                // Update currently targeted object
                //my code here
                TargetedObject = newTargetedObject;<br>                if ( == "Player1")<br>                {<br>                    //Debug.Log("selected");<br>                    TargetedObject.SendMessage("OnSelect");<br>                } 
                if (TargetedCursorModifier != null)
                    TargetedCursorModifier.GetModifiedTransform(this, out targetPosition, out targetRotation, out targetScale);
                    // If no modifier is on the target, just use the hit result to set cursor position
                    targetPosition = focusDetails.Point + (lookForward * SurfaceCursorDistance);
                    Vector3 lookRotation = Vector3.Slerp(focusDetails.Normal, lookForward, LookRotationBlend);
                    targetRotation = Quaternion.LookRotation(lookRotation == ? lookForward : lookRotation, Vector3.up);

            float deltaTime = UseUnscaledTime
                ? Time.unscaledDeltaTime
                : Time.deltaTime;

            // Use the lerp times to blend the position to the target position
            transform.position = Vector3.Lerp(transform.position, targetPosition, deltaTime / PositionLerpTime);
            transform.localScale = Vector3.Lerp(transform.localScale, targetScale, deltaTime / ScaleLerpTime);
            transform.rotation = Quaternion.Lerp(transform.rotation, targetRotation, deltaTime / RotationLerpTime);

Save the code and close the Visual Studio.

Next, go to Unity Editor and Run the project. You will notice that the cube called Player1 will change its color to red. Right click to move around the mouse, and change the color of the player1. When the Gaze is out of the Player1, the color will be original color - otherwise, it will be red.

Handling Tap / Input Click Event

Now we going to add a Tap event to the app. By taping at the gaze object, we going to shoot a bullet at it. The bullet will be a small sphere with a force added to it. The forward direction will be calculated based on the camera forward look.

  • Go to the Cursor.cs again and double click to open it in Visual Studio
  • Locate the OnInputClicked event in the cs file
  • Copy paste the full code from below to the function
 public virtual void OnInputClicked(InputClickedEventData eventData)
            // Open input socket for other cool stuff...
            var ball = GameObject.CreatePrimitive(PrimitiveType.Sphere);
            ball.transform.localScale = new Vector3(0.1f, 0.1f, 0.1f);
            var rigid = ball.AddComponent();
            rigid.mass = 0.5f;
            var trans = MixedRealityCameraManager.Instance.transform;

            rigid.position = trans.position;
            var transformed = trans.forward;
            transformed = Quaternion.AngleAxis(-10, trans.right) * transformed;
            rigid.AddForce(transformed * 500f);
            Debug.Log("Input Transformed");

In the above code, we are creating the sphere at run time and setting it size to small. Then, we’re adding force to throw in a forward direction. Save and close Visual Studio.

Now run the app in the Unity Editor. To throw the spheres, tap (left mouse click) while holding down the Shift Key. To move around, hold the right mouse down. 

Running the code on Emulator

Windows 10 FCU comes with the Mixed Reality portal for testing the app in the emulator. Go to settings to enable the developer mode, and then enable Mixed Reality

Follow this link here to setup the portal:

Run the Mixed Reality Portal and enable the headset from the For Developer option Once enabled, the Mixed Reality will be executed.

Once the Portal is running, now go back to Unity Editor and run the project. This time it will execute inside the Mixed Reality portal and also in Unity. You can see it in both places. Play around and enjoy!

Senthamil Selvan is a Windows Development MVP. He has published several apps to the windows store. He is passionate about Microsoft Apps and HoloLens development. Microsoft SharePoint is his profession, and App development is his passion. Follow him on Twitter @altfo.