Joint Orientation

Kinect for Windows 1.5, 1.6, 1.7, 1.8

Starting in SDK 1.5, Kinect for Windows provides joint orientation information for the skeletons tracked by the ST pipeline. The bone orientation is provided in two forms:

• A hierarchical rotation based on a bone relationship defined on the skeleton joint structure
• An absolute orientation in Kinect camera coordinates

The orientation information is provided in form of quaternions and rotation matrices for use in different animation scenarios.

A sample application using joint orientation for avateering is provided in Kinect for Windows Developer Toolkit 1.5.0.

The sample is called "Avateering"; beyond providing an example of using hierarchical joint orientation for animating an avatar, it provides a content retargeting pipeline with different stages designed to smooth the raw signal of the skeletal tracking to provide a smoother animation of the avatar.

How to access joint orientation in C#

In managed code, access the joint orientation using the BoneOrientations property, which gives access to the BoneOrientationCollection and the BoneOrientation class.

  private void DrawSkeletonsWithOrientations()
  {
      foreach (Skeleton skeleton in this.skeletonData)
      {
          if (skeleton.TrackingState == SkeletonTrackingState.Tracked)
          {
              foreach (BoneOrientation orientation in skeleton.BoneOrientations)
              {
                  // Display bone with Rotation using quaternion
                  DrawBonewithRotation(orientation.StartJoint, orientation.EndJoint, orientation.AbsoluteRotation.Quaternion);
                  // Display hierarchical rotation using matrix
                  DrawHierarchicalRotation(orientation.StartJoint, orientation.HierarchicalRotation.Matrix)
              }
          }
      }
  }

How to access joint orientation in C++

In native mode, you call the NuiSkeletonCalculateBoneOrientations API that fills an array of NUI_SKELETON_BONE_ORIENTATION structures.

  void MyApplication::DrawSkeletonsWithOrientations(NUI_SKELETON_FRAME* pSkeletonFrame)
  {
    for (int i = 0; i < NUI_SKELETON_COUNT; i++)
    {
      const NUI_SKELETON_DATA & skeleton = pSkeletonFrame->SkeletonData[i];
 
      NUI_SKELETON_BONE_ORIENTATION boneOrientations[NUI_SKELETON_POSITION_COUNT];
      NuiSkeletonCalculateBoneOrientations(&skeleton, boneOrientations);

      if (skeleton.eTrackingState == NUI_SKELETON_TRACKED)
      {
        for (int j = 0; j < NUI_SKELETON_POSITION_COUNT; j++)
        {
          NUI_SKELETON_BONE_ORIENTATION & orientation = boneOrientations[j];
                    
          // Display bone with Rotation using quaternion
          DrawBoneWithRotation(
              orientation.startJoint,
              orientation.endJoint,
              orientation.absoluteRotation.rotationQuaternion);
 
          // Display hierarchical rotation using matrix
          DrawHierarchicalRotation(
              orientation.startJoint,
              orientation.hierarchicalRotation.rotationMatrix);
        }
      }
    }
  }

Bones Hierarchy

We define a hierarchy of bones using the joints defined by the skeletal tracking system.

The hierarchy has the Hip Center joint as the root and extends to the feet, head, and hands:

Figure 1.  Joint Hierarchy

Bones are specified by the parent and child joints that enclose the bone. For example, the Hip Left bone is enclosed by the Hip Center joint (parent) and the Hip Left joint (child).

Bone hierarchy refers to the ordering of the bones defined by the surrounding joints; bones are not explicitly defined as structures in the APIs. Bone rotation is stored in a bone’s child joint. For example, the rotation of the left hip bone is stored in the Hip Left joint.

Hierarchical Rotation

Hierarchical rotation provides the amount of rotation in 3D space from the parent bone to the child. This information tells us how much we need to rotate in 3D space the direction of the bone relative to the parent.

This is equivalent to considering the rotation of the reference Cartesian axis in the parent-bone object space to the child-bone object space, considering that the bone lies on the y-axis of its object space.

Figure 2.  Hierarchical Bone Rotation

Absolute Player Orientation

In the hierarchical definition, the rotation of the Hip Center joint provides the absolute orientation of the player in camera space coordinates. This assumes that the player object space has the origin at the Hip Center joint, the y-axis is upright, the x-axis is to the left, and the z-axis faces the camera.

Figure 3.  Absolute Player Orientation is rooted at the Hip Center joint

To calculate the absolute orientation of each bone, multiply the rotation matrix of the bone by the rotation matrices of the parents (up to the root joint).

Accessing hierarchical orientation in C#

The HierarchicalRotation property returns a BoneRotation object that has a rotation matrix and a quaternion vector.

  boneOrientations[JointType.HipRight].HierarchicalRotation

Accessing hierarchical orientation in C++

The NUI_SKELETON_BONE_ORIENTATION structure returns a NUI_SKELETON_BONE_ROTATION structure that has a rotation Matrix4 and a quaternion Vector4.

    
    boneOrientations[NUI_SKELETON_POSITION_HIP_RIGHT].hierarchicalRotation 

Seated Mode Orientation

When tracking a skeleton in seated mode, the root of the joints becomes the Shoulder Center joint. The rotation for the non-tracked joints will be "no rotation". (This is equivalent to the identity matrix.)

Absolute Orientation

Absolute orientation provides the orientation of a bone in 3D camera space. The orientation of a bone is relative to the child joint and the Hip Center joint still contains the orientation of the player. Same rules for seated mode and non-tracked joints applies.

Also in this case, the orientation of a bone is stored in relation to the child joint and the Hip Center joint still contains the orientation of the player.

These rules apply to seated mode and non-tracked joints also.

Accessing absolute orientation in C#

The AbsoluteRotation property returns a BoneRotation.

  boneOrientations[JointType.HipRight].AbsoluteRotation

Accessing absolute orientation in C++

The NUI_SKELETON_BONE_ORIENTATION structure returns a NUI_SKELETON_BONE_ROTATION structure that has a rotation Matrix4 and a quaternion Vector4.

  boneOrientations[NUI_SKELETON_POSITION_HIP_RIGHT].absoluteRotation
        

Orientation Calculation

We calculate the orientation of the bones based on the positions of the joints returned by the skeletal tracking pipeline.

If the positions of the joints change in a managed code application, the runtime performs an automatic orientation re-calculation of the bones.

Note

If you are trying to adjust the positions of the joints and then access the orientations in managed code, the calculation will take place every time you access the orientation. You might want to adjust all the joint positions before accessing the orientation.

Avatar retargeting and smoothing

Joint orientation information is based solely on the skeletal tracking joints; there are no specific avatar retargeting calculations made in the library, given its generic purpose.

To see an example of smoothing joints and retargeting the skeletal tracking signal for avatar animation, see the Avateering-XNA sample.

Smoothing can be applied to skeletal joints before calculating their orientation; use the smoothing algorithm provided in the runtime or change the joints manually based on your own smoothing algorithm.