Training and procedural guidance powered by mixed reality

Solution Idea

If you'd like to see us expand this article with more information (implementation details, pricing guidance, code examples, etc), let us know with GitHub Feedback!

Enable your team and employees to learn new processes and materials faster, with fewer errors, and greater confidence by providing persistent holographic instructions mapped to precise locations in their physical workspace. Jumpstart employee comprehension with head-up, hands-free experiences using HoloLens devices. And with Azure Spatial Anchors, you can place directions on the procedure's most important objects and return to this content over time.

Architecture

Architecture Diagram Download an SVG of this architecture.

Data Flow

  1. The user creating the training session authenticates using their Azure Active Directory credentials from HoloLens.
  2. The client application connects to its own web service to create a training session. Metadata about that training session is stored in Azure Cosmos DB.
  3. The user scans the environment and places a first anchor where the first step of the procedure needs to happen. Azure Spatial Anchors validates that the user has sufficient permissions to create anchors via Azure AD, and then stores the anchor.
  4. The user records a video of the procedure on HoloLens and uploads it to Azure
  5. The video is encoded with Media Services and prepared for on-demand viewing, as well as processed with Video Indexer for better content search. Video Indexer stores the metadata on Azure Cosmos DB.
  6. The app saves against its web service the anchor ID for that first step, alongside a link to the video.
  7. The user, in the same session, then moves on to step 2, places an anchor there, and again records a video of the procedure and saves the resulting anchor ID and video link to its web service. That process is then repeated until all steps in the procedure are executed. As the user moves from step to step, previous anchors are still visible with their respective step number.
  8. A trainee comes in, selects the training session, retrieves anchor IDs and links to videos that are part of the procedure.
  9. The trainee scans the room to find the anchors indicating the real-world location of each step in the procedure. As soon as one is found, all anchors are retrieved and shown in the app.
  10. The trainee can then retrace the exact steps of the expert who recorded the procedure, and view holographic videos of each step at the right location in the lab.

Components

Next steps