Introduction to Gestures in Unity®

This tutorial will teach you how to use the Project Prague toolkit for Unity in order to add gestures to a Unity application. We will create a simple Unity project that generates and destroys 3D-primitives every time the user executes certain gestures. We will then demonstrate how to use the CameraGesturesController prefab from our Unity toolkit in order to move the main scene camera with gestures.

This tutorial assumes you have basic familiarity with the C# programming language. We do not assume familiarity with the Unity environment.

This tutorial will take approximately 30 minutes to complete.

Download the Final Result

The final Unity project obtained in this tutorial can be found in our open-source samples repository on GitHub. After you clone the repository, follow these steps to run the application:

1. Launch Unity, in the Projects tab select Open.
2. Browse to the Unity\Tutorials\Introduction directory within the cloned repository.
3. Press the play button (or Ctrl+P) to run the scene.

Prerequisites

Make sure you own one of our supported depth cameras and that your PC meets the Project Prague requirements. Once you have all your hardware in place, set up Project Prague on your machine.

You will also need the following software tools for this tutorial

• Unity editor version 5.5 or higher.
• Visual Studio 2015 or later with Visual Studio Tools for Unity.

Refer to our FAQ page for help with installing Visual Studio Tools for Unity.

Step 1 - Create and Configure a New Unity Project

1. Launch the Unity editor and create a new project in a location of your choice:

   

   Name the project Introduction and select a location where you would like to place it:

   

2. Clone our GitHub samples repository, and import our toolkit for Unity to your project - from the Assets menu, select Import Package and Custom Package...:

   

   Browse to the package location on your disk (Unity directory within the cloned repository) and open it:

   

   Choose to Import all the package contents:

   

3. This is a good time to save your scene - press Ctrl+S and give your scene a name:

   

Tip

It's a good practice to save the scene from time to time to avoid losing your work. We recommend you save the scene (using Ctrl+S) at the end of every tutorial step.

Step 2 - Connecting to the Gestures Service

We will now establish a connection between our Unity application and the Gestures Service. Please make sure that your depth camera is connected to the computer and that the Gestures Service is running (launch it using the Microsoft.Gestures.Service shortcut that was created on your desktop during Project Prague installation).

1. In the Project window of the Unity editor, browse to the Prefabs directory (under MicrosoftGesturesToolkit):

   

   We will use the GesturesManager prefab in order to communicate with the Gestures Service. Every Unity scene that works with Project Prague must contain exactly one instance of the GesturesManager.

   We will also use the UIManager prefab to display useful information about the state of connection to the Gestures Service.

   Drag and drop the corresponding prefabs to your scene:

   

2. Now play the scene by pressing the play button or using the Ctrl+P keyboard shortcut. Make sure that the icon in the bottom right corner is in the Detecting state:

   

   This indicates that our application was able to establish a connection with the Gestures Service and that gesture detection is active.

   If, instead, the reported state is Disconnected:

   

   Please launch your Gestures Service (use the Microsoft.Gestures.Service desktop shortcut). The state should change to Detecting within several seconds of launching.

   If the reported state is Connected:

   

   Please make sure that the Enable Detection checkbox in the Microsoft Gestures Service window is checked:

   

Important

Don't forget to exit the play mode (by pressing the play button or Ctrl+P again) before you move to the next step. Any changes you make to the scene while it is playing will be discarded once you exit play mode.

Step 3 - Creating a Script to Generate 3D-Primitives in the Scene

To demonstrate how gestures can be used in a Unity game, we will start by adding a new game object and associating it with a C#-script. We will program the script to generate a new 3D-primitive in the scene on every left mouse button click. On the next step, we will specify a gesture and configure it to trigger the primitive-generating code in our script.

1. Use the GameObject menu to Create Empty, rename the new object to PrimitiveFactory:

   

   In the Project window, create a new folder under Assets and name it Scripts. This folder will contain the code which will be executed upon gesture detection.

   

   In the Scripts folder, create a new script named PrimitiveFactory (no need to add the .cs extension to the script name):

   

2. To associate the PrimitiveFactory script with the PrimitiveFactory game object, drag and drop the script to the blank area below the Add Component button in the game object's Inspector window:

   

   In the Project window, double click the PrimitiveFactory icon to edit the script in Visual Studio. Once open, replace the contents of PrimitiveFactory.cs with the following code:

   Warning

   It looks like the sample you are looking for has moved! Rest assured we are working on resolving this.

3. Play the scene and use your mouse pointer to left-click anywhere within the scene frame. Every click will instantiate a new randomly generated 3D-primitive:

   

Tip

Take a look at the Console window of the Unity editor. You will find there various messages, warnings and errors, generated by the various entities in our scene and the Unity platform itself.

Step 4 - Using a Gesture to Generate New 3D-Primitives in the Scene

On this step, we will create the necessary wiring so that you can use a gesture to trigger the 3D-primitive generation.

1. The GestureTrigger prefab provided with our Unity toolkit allows you to designate a method in your project to be executed whenever some gesture is detected.

   Drag and drop the GestureTrigger prefab from the Project window to your scene in the Hierarchy window:

   

2. Select the GestureTrigger game object in the Hierarchy window, and turn to the Inspector window:

   

   The sections of the UI (user interface), numbered 1 to 4 above, allow you to configure a gesture and its effect:

   
   

   1	Select in which way you would like to specify your gesture - either pick a gesture from a list of predefined "stock" gestures or specify a custom gesture in XAML language.
   2	Specify which method is to be executed when the gesture is detected.
   3	Specify which method is to be executed when a certain state in the gesture state-machine is detected.

   Visit our overview page to read more about the concept and structure of a gesture in Project Prague.

   In this tutorial we will focus on sections 1 and 2. Namely, we will wire the predefined "Tap" stock-gesture to trigger the CreateRandomPrimitive() method of the PrimitiveFactory game object. Using XAML to specify gestures and section 3 of the GestureTrigger UI will be covered in the 3D Object Manipulation tutorial.

3. Make sure GestureTrigger is still the selected object in the Hierarchy window and choose the Tap gesture from the Stock Gesture drop-down list in the Inspector window:

   

4. To make the methods of PrimitiveFactory available for GestureTrigger, click the + sign in the On Trigger () pane, then drag and drop the PrimitiveFactory game object from the Hierarchy window to the None (object) box in the Inspector:

   

   Finally, to designate CreateRandomPrimitive() as the handler method for the Tap gesture, click the No Function drop-down in the Inspector window of GestureTrigger, select PrimitiveFactory and CreateRandomPrimitive ():

   

5. Play the scene now. You should still be able to generate a new 3D-primitive using the left mouse button. In addition, you should now be able to do that with the Tap gesture (perform with either hand):

   

   If you cannot successfullly use the Tap gesture to generate new primitives in the scene, please refer to the troubleshooting section.

Step 5 - Using a Gesture to Destroy all 3D-Primitives in the Scene

On this step, we will add a gesture to clear all previously generated 3D-primitives from the scene.

1. First, add a method named DestroyAllPrimitives() to our PrimitiveFactory script (PrimitiveFactory.cs should still be open in Visual Studio):

   Warning

   It looks like the sample you are looking for has moved! Rest assured we are working on resolving this.

2. Before we wire the new method to a gesture, we will test it using the right mouse button. Replace the existing Update() method in the PrimitiveFactory script with the following code:

   Warning

   It looks like the sample you are looking for has moved! Rest assured we are working on resolving this.

3. Play the scene. Create several primitives by clicking on the left mouse button. Destroy all primitives by clicking the right button.

4. We will now wire the Finger Snap gesture to invoke the DestroyAllPrimitives() method, like we did in step 4:

   • Add another GestureTrigger game object to the scene.
   • In the Inspector view of the new GestureTrigger game object, select the Finger Snap gesture from the Stock Gesture drop-down list.
   • In the On Trigger () pane, create a new element by pressing + sign. Drag and drop the PrimitiveFactory game object to the None (Object) box to associate it with the new On Trigger () element.
   • Finally, click on the No Function drop-down list and select the DestroyAllPrimitives() of the PrimitiveFactory object.

5. Our Discovery Client demo application uses the Finger Snap gesture to open the start menu. Note that we are also using this gesture in our Unity application. You should disable or close the Discovery Client because we don't want Finger Snap to have any side-effects outside of Unity. To disable - click on the Project Prague tray icon and choose Disable:

   

   You can always follow the same steps to re-enable the Discovery Client.

6. Now run the scene. Generate a few primitives using the Tap gesture and destroy them using the Finger Snap gesture (perform with either hand):

   

Step 6 - Using a Gesture to Control the Camera

On this step we will use a Project Prague toolkit prefab to control the position of the main camera in the scene with gestures.

1. In the Project window, locate the CameraGesturesController prefab under the MicrosoftGesturesToolkit\Prefabs directory. Drag and drop it to your scene:

2. Play the scene. Now, in addition to creating and destroying objects, you can control the camera using the following gesture (perform with either hand):

   

   Clinching your hand into a fist (InitSpreadPose → GrabPose transition) will cause the camera to "snap" to the hand - allowing you to tumble and dolly the camera by moving your fist through space.

   Releasing the clinch by spreading your fingers apart (GrabPose → FinalSpreadPose transition) will cause the camera to stop tracking your hand.

3. If you are curious, examine the CameraGesturesController game object in the Inspector window. Note that it has a Gesture Trigger component. We used this component to wire the different states in the GrabReleaseGesture to various methods in the CameraManipulation script (found in MicrosoftGesturesToolkit\Scripts directory).

   To learn more about specifying custom gestures with XAML and wiring their states to methods in a game object, refer to step 3 of the 3D Object Manipulation tutorial.

Troubleshooting

If you've reached the end of Step 4 and the Tap gesture is not working for you, please play the Unity scene, open the Microsoft Gestures Service window and verify the following:

1. When you bring your hand close to the depth camera, you can see it in the image section of the Microsoft Gestures Service window, with colorful arrows indicating the estimated positions of the fingers and palm. If you don't see such an image, you are probably experiencing camera issues - please refer to our camera troubleshooting page.
2. Unity appears in the list of Clients and Unity_TapGesture appears in the sub-list of gestures associated with the Unity client. If this is not the case, please verify that
   • The icon in the bottom right corner of your Game window in Unity displays the Detecting state (see step 2).
   • You have selected the correct gesture, Tap, in the GestureTrigger user interface (see step 4).
3. When the Unity window is in the foreground and you execute the Tap gesture, the TimeLine associated with Unity_TapGesture should advance. If this is not the case, you are probably not executing the gesture correctly - make sure to perform the Tap motion with your thumb and index fingers only.

If you still cannot use the Tap gesture to generate new 3D-primitives in the scene, please leave a comment on this page, and we will do our best to provide help.