How do I track multiple bodies with Azure Kinect for Unity?

Alex Lamarche 0

With the unity example scene for the azure Kinect, I've been able to set up single skeleton tracking and create a few demo applications. Now I want to track and manage multiple bodies, but modifying the sample code only gets me so far.

My avatars will glitch and flicker to various positions when I have more than one tracker set up. I suppose that the bodies are being tracked, but not "memorized," that is, the body tracker only sees bodies and not the movement of bodies across multiple frames.

I've cloned this repo: https://github.com/microsoft/Azure-Kinect-Samples.git

I followed this tutorial: https://learn.microsoft.com/en-us/shows/mixed-reality/azure-kinect-body-tracking-unity-integration

In short: All I want is to have character A mapped to myself and character B mapped to my colleague next to me. The problem is that the characters will flicker between our positions. Can anyone help me understand how to track multiple bodies in unity with the Azure Kinect dk and unity plugin?

LeelaRajeshSayana-MSFT 13,621 Reputation points

2023-11-02T22:31:52.95+00:00

Hi @Alex Lamarche Greetings! Welcome to Microsoft Q&A forum. Thank you for posting this question here.

I am checking to see if you have reviewed the documentation on Access data in body frame. The article suggests that if we are tracking multiple bodies, we can get the details on each body being tracking by iterating through k4abt_frame_get_num_bodies() function. Please refer the code sample provided in the article and see if that helps.
Alex Lamarche 0 Reputation points

2023-11-03T00:53:11.45+00:00
Hey thanks for the timely response!

I iterate through all of the bodies. However, when I check the order in which the body ids appear, they tend to flip around and "flicker" which causes the rendered avatars to also flicker between the body positions:

Ex of 3 bodies in camera view
frame 1: [1, 2, 3, 0, 0, 0, 0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0],
frame 2: [3, 1, 2, 0, 0, 0, 0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0],
frame 3: [3, 2, 1, 0, 0, 0, 0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0 ,0],
. . .
where the array represents the order of body ids in currentFrame.Bodies:Body[]

I'm trying to find out how to maintain a reference to the same body throughout multiple frames. Tomorrow when I get back to my code, I can show you my sample. I can also post a GIF of my problem to better illustrate.

My current idea for a solution is:

Track all positions of skeleton bodies for current frame

Compare the last frames skeletons' positions to the current frames skeleton positions

Adjust the id references based on relative distances between frames

Render respective avatars to previously tracked bodies/skeletons.

This approach - I would hope - keeps track of the bodies and avatars based on motion

I hope that all makes sense
Alex Lamarche 0 Reputation points

2023-11-03T14:03:39.32+00:00

@LeelaRajeshSayana-MSFT Here is what I'm talking about:

For some reason I cannot get the quality to be higher, but the gif illustrates my problem. The output logs are similar to the logs described in my previous comment.

Also, I know it doesn't look like it, but the positions will swap between each other. The Kinect doesn't lose tracking with only a single person, so this is less of a position accuracy problem and more of a specific body tracking problem.
LeelaRajeshSayana-MSFT 13,621 Reputation points

2023-11-15T00:03:07.84+00:00

Hi @Alex Lamarche Please post this question to GitHub issues page You might get assistance from the product team. Kindly note that there might be delays in getting a response as the GitHub repository is not actively maintained.