MR Learning Base Module - 3D Object Interaction

In this lesson, we will go through basic 3D content and user experience. We will learn how to organize 3D objects as part of a collection, learn about bounding boxes for basic manipulation, learn about near and far interaction, and learn about touch and grab gestures with hand tracking.

Objectives

  • Learn how to organize 3D content using MRTK's Grid Object Collection
  • Implement bounding boxes
  • Configure 3D objects for basic manipulation (move, rotate, and scale)
  • Explore near and far interaction
  • Learn about additional hand tracking gestures such as grab and touch

Instructions

Organizing 3D Objects in a Collection

  1. Right click on your hierarchy and select, “create empty.” This creates an empty game object. Rename it to “3DObjectCollection.” This is where we will put all of our 3D Objects. Make sure the collection’s positioning is set at x = 0, y = 0, and z = 0.

Lesson4 Chapter1 Step1im

  1. Import BaseModule Assets Using the same instructions to import custom packages outlined in Lesson1. The BaseModule assets include 3D modules and other useful scripts that will be used throughout this tutorial. The BaseModule unity package can be found here: https://github.com/Microsoft/MixedRealityLearning/releases/tag/V1.1

  2. The coffee cup prefab can be recognized by a blue cube next to it. Do not select the coffee cup with the blue cube and small white paper (that denotes the original 3D model and not the prefab.)

Lesson4 Chapter1 Noteaim

  1. Drag the coffee cup prefab of your choice into the "3DObjectCollection" game object from step 1. The coffee cup is now a child of the collection.

Lesson4 Chapter1 Step4ima

  1. Next, we'll add more 3D objects into our scene. Below is a list of objects we are going to add in this example. As you add the objects you may find that they appear in your scene in various sizes. Adjust the scale of each 3D model under the transform setting in the inspector panel. Recommended adjustments for this example are listed with the objects below. Search these words in the search box in your project panel and drag the 3D object prefab into the "3DObjectCollection" object similar to the previous step. You will find these collection of prefabs in Assets>BaseModuleAssets>Base Module Prefabs
  • Search for “TheModule_BaseModuleIncomplete.” Drag into scene. Set the scale to x = 0.03, y = 0.03, z = 0.03.
  • Search for “Octa_BaseModuleIncomplete.” Drag into scene. Set the scale to x = 0.13. y = 0.13, z =0.13.
  • Search for “EarthCore_BaseModuleIncomplete.” Drag into scene. Set the scale to x = 50.0 y = 50.0, z = 50.0.
  • Search for “Cheese_BaseModuleIncomplete.” Drag into scene. Set the scale to x = 0.05, y = 0.05, z = 0.05.
  • Search for “Model_Platonic_BaseModuleIncomplete.” Drag into scene. Set the scale to x = 0.13, y = 0.13, z = 0.13.
  • Search for "CoffeeCup_BaseModuleIncomplete." Drag into scene.

Lesson4 Chapter1 Step5im

  1. Add 3 cubes into your scene. Right click the “3DObjectCollection” object, select “3D Object”, then select “Cube.” Set the scale to x = 0.14, y = 0.14, and z = 0.14. Repeat this step 2 additional times to create a total of 3 cubes. Alternatively, you may duplicate the cube twice for a total of 3 cubes. You may also choose to use the three prepared cube prefabs from Assets>BaseModuleAssets>Base Module Prefabs and select GreenCube_BaseModuleIncomplete, BlueCube_BaseModuleIncomplete and OrangeCube_BaseModuleIncomplete.

Lesson4 Chapter1 Step6im

  1. Organize your collection of objects to form a grid using the procedure described in Lesson 2 using the MRTK’s Grid Object Collection. Refer to the image below to see an example of configuring the objects in a 3x3 grid.

Lesson4 Chapter1 Notebim

Note: You may notice that some of the objects are off-center, such as the objects in the image above. This is because prefabs or objects may have child objects that are not aligned. Feel free to make any necessary adjustments to object positions or child object positions to achieve a well-aligned grid.

Manipulating 3D Objects

  1. Add the ability to manipulate a cube. To add the ability to manipulate 3D objects you must do the following:
  • Select the 3D object you want to manipulate in your hierarchy (in this example, one of your cubes).
  • Click “add component.”
  • Search for “manipulation.”
  • Select “manipulation handler.”
  • Repeat for all 3D objects under the “3DObjectCollection” object but not the “3DObjectCollection” itself.
  • Ensure all 3D objects have a collider or box collider (Add Component > box collider).

Lesson4 Chapter2 Step1im

The manipulation handler is a component that will allow you to adjust settings for how objects behave when being manipulated. This includes rotation, scaling, moving, and constraining movement on certain axes.

  1. Restrict one cube so that it can only be scaled. Select one cube in the “3DObjectCollection” object. In the inspector panel, next to “two handed manipulation type,” click the drop-down menu and select “scale.” This makes it so that the user can only change the cube’s size.

Lesson4 Chapter2 Step2im

  1. Change the color of each cube so that we can differentiate between them.
  • Go to the project panel and scroll down until you see “MixedRealityToolkit.SDK” then select it.
  • Select the “Standard Assets” folder.
  • Click on the “materials” folder.
  • Drag a different material onto each of your cubes.

Note: You can choose any color for your cubes. For our example, we are going to use “glowingcyan,” “glowingorange”, and "green." Feel free to experiment with different colors. To add the color to the cube, click the cube you want to change the color of, then drag the material to the mesh renderer's material field in the cube's inspector panel.

Lesson4 Chapter2 Step3im

  1. Select another cube in the “3DObjectCollection” object and make it so that its movement is constrained to the fix distance from the head. To do this, on the right of “constraint on movement,” click on the dropdown menu and select “fix distance from the head.” This makes it so that the user can only move the cube within their field of vision.

Lesson4 Chapter2 Step4im

Goal of the next few steps: We will enable grab and interaction with our 3D objects. We will apply different manipulation settings

  1. Select the cheese object and in the inspector panel, click “add component.”

  2. Search in the search box for “Near Interaction Grabbable” and select the script. This component allows users to reach out and grab the objects with tracked hands. Objects will also be allowed to be manipulated from a distance, unless the "Allow Far Manipulation" checkbox is unchecked (denoted by green circle in image below.)

Lesson4 Chapter2 Step6im

  1. Add “Near Interaction Grabbable” to the Octa object, Platonic object, earth core, lunar module, and coffee cup by repeating step 5 and 6 on those objects.

  2. Remove the ability of far manipulation from the Octa object. To do this, select the Octa in the hierarchy, and uncheck the “allow far manipulation” checkbox (marked by a green circle). This makes it so that users can only interact with the octa directly using tracked hands.

Note: For the full documentation of the manipulation handler component and it's associated settings, please refer to the MRTK Documentation.

  1. Ensure that the "Near Interaction Grabbable" component has been added to the earth core, the lunar module and the coffee cup (see step 7).

  2. For the lunar module, change the Manipulation Handler settings so that it rotates about the object's center for both near and far interaction, as shown in the image below.

Lesson4 Chapter2 Step10im

11: For the earth core, change the release behavior to “nothing.” This makes it so that once the earth core is released from the users’ grasp, it doesn’t continue to move.

Lesson4 Chapter2 Step11im

Note: This setting is useful for scenarios such as creating a ball that you can throw. Keeping the velocity and the angular velocity makes it so that once the ball is released it will continue to move at the velocity it was released at, similar to how a physical ball would behave.

Adding Bounding Boxes

Bounding boxes make it easier and more intuitive to manipulate objects with one hand for both direct manipulation (near interaction) and ray-based manipulation (far interaction.) Bounding boxes offer "handles" that can be grabbed for scaling and rotating objects along specific axes.

Note: Before you can add a bounding box to an object you first need to have a collider on the object (e.g., a box collider.) As we did previously in this lesson. Colliders can be added by selecting the object, and in the object's inspector panel selecting Add Component>Box Collider.

  1. Add a box collider to the earth core object, if one does not already exist (box collider and setup not required if using the prefab provided in the Base Module Assets folder, per the instructions given.) In the case of the earth core, we will need to add the box collider to the "node_id30" object underneath the earth core, as shown in the image below. select node_id30 and in the object's inspector tab, click “add component” and search for “box collider.”

Lesson4 Chapter3 Step1im

Lesson4 Chapter3 Step2im

Note: Make sure that you visualize the box collider so that it’s not too big or too small. It should be roughly the same size as the object it’s surrounding (in this example, the earth core). Adjust the box collider as needed by selecting the edit collider option in the box collider. You can either changing the x, y, and z values or drag the bounding box handlers in the editor scene window.

Lesson4 Chapter3 Noteim

  1. Add a bounding box to the earth core's "node_id30" object. To do this, select the "node_id30" object from the "3DObjectCollection." In the inspector tab, click “add component” and search for “bounding box.” Ensure that the bounding box, box collider, and manipulation scripts (manipulation handler, near interaction grabbable) are all on the same game object.

  2. In the bounding box's "Behavior" section, select “activate on start” from the Activation dropdown list. To review additional details regarding the various activation options and other bounding box options, please see the MRTK's bounding box documentation

In the next few steps, we will also change how the bounding box looks by adjusting the default box material, the material while it’s being grabbed, as well as the visualization of handles (corner and side handles). The MRTK contains several options to customize the bounding box.

  1. In the project panel, search for “boundingbox” and you’ll see a list of materials denoted by a blue sphere in the search results, as shown in the image below.

  2. Drag the “boundingbox” material into the box material slot on the bounding box component. Also grab the “boundingboxgrabbed” material and put that in the box grabbed material slot on the bounding box component.

  3. Drag the “MRTK_BoundingBox_ScaleWidget” material into the scale handle prefab slot on the bounding box component.

  4. Drag the “MRTK_BoundingBox_RotateWidget” material into the rotation handle slot on the bonding box component.

Lesson4 Chapter3 Step4 7Im

  1. Make sure the bounding box is targeting the right object. In the bounding box component, there is the “target object” and “bounds override” scripts. Make sure to drag the object that has the bounding box around it to both of these slots. In this example, drag the "node_id30" object to both these slots, as shown in the image below.

When you start or play the application, your object will now be surrounded by a blue frame. You’re welcome to drag the corners of that frame to resize the object. If we want the scaling handles and the rotation handles to be larger and more visible, we recommend using the default bounding box settings (avoiding steps 4 - 7.)

Lesson4 Chapter3 Step8im

  1. To return to the default bounding box visualization, in the inspector panel of the bounding box's object, select the rotation handle prefab and press the delete key, you will now see a bounding box visualization similar to the image below. Note: the bounding box visualizations only appears when in play mode.

Lesson4 Chapter3 Step9im

Adding Touch Effects

In this example, we are going to play a sound effect when you touch an object with your hand.

  1. Add an audio source component to your game object. Select the "octa" object in your scene hierarchy. In the inspector panel, click the "add component" button, search for and select "audio source." We’ll use this audio source to play a sound effect in a later step.

Note: Ensure that the "Octa" object has a box collider on it.

  1. Add the “near interaction touchable” component. Click the "Add Component" button in the inspector panel and search for “near interaction touchable.” Select it to add the component. NOTE: fix screenshot to highlight that we're adding the component, and not just highlighting box collider.

Note: Previously we added the “near interaction grabbable.” The difference between this and “near interaction touchable” is that the "grabbable" interaction is intended for an object to be grabbed and interacted with. The "touchable" component is intended for the object to be touched. Both components can be used together for a combination of interactions.

Lesson4 Chapter4 Step1 2Im

  1. Add in the “hand interaction touch” script. Note that this script is included with the unity scene you imported as part of this demo package and it is not included in the original MRTK. Just like the previous step, click “add component” and search for “hand interaction touch” to add it. Notice that you have 3 options with the script:

    • "On touch completed." This will trigger when you touch and release the object.
    • "On touch started." This will trigger when the object is touched.
    • "On touch updated." This will periodically trigger while your hand is touching the object.

    For this example, we will be working with the “on touch started” setting.

  2. Click the “+” button on the “on touch started” option, as shown in the image below. Drag the octa object into the empty field.

  3. In the drop-down that says "no function" (above green rectangle in the image below), select AudioSource> PlayOneShot. We will add an audio clip to this field using the concepts below:

    • The MRTK does provide a small list of audio clips. Feel free to explore these in your project panel. You will find them under the “MixedRealityToolkit.SDK” folder and then the “standard assets” folder. There you will see an “audio” folder where all the audio clips are.
    • For this example, we are going to use the "MRTK_Gem" audio clip.
    • To add an audio clip, simply drag the clip you want from the project panel into the AudioSource.PlayOneShot (marked by green box in the example above) in the inspector panel.

    Now when the user reaches out and touches the octa object, the audio track “MRTK_Gem” will play. The "Hand Interaction Touch" script will also adjust the color of the object when touched.

Lesson4 Chapter4 Step3 5 Noteim

Congratulations

In this lesson, you learned how to organize 3D objects in a grid collection and how to manipulate 3D objects (scaling, rotating, and moving) using near interaction (directly grabbing with tracked hands) and far interaction (using gaze rays or hand rays.) You also learned how to put bounding boxes around 3D objects and learned how to use and customize the gizmos on the bounding boxes. Finally, you learned how to trigger events when touching an object.

Next Lesson: Advanced Input