Development overview

Mixed reality apps can be developed using a variety of developer technologies. HoloLens runs apps that are built with the Universal Windows Platform. Immersive headsets run Universal Windows Platform apps as well as Win32 applications. By getting familiar with middleware tools like Unity, you can start building mixed reality experiences today. Leverage the open source Mixed Reality Toolkit to get started quickly. Mixed reality services, such as Azure Spatial Anchors, have SDKs that can integrate into various cross-platform developer technologies as well.

Basics of mixed reality development

Mixed reality experiences are enabled by new Windows features for environmental understanding. These enable developers to place a hologram in the real world, and allow users to move through digital worlds by literally walking about.

These are the core building blocks for mixed reality development:

Input HoloLens (1st gen)HoloLens 2 Immersive headsets
Head gaze✔️✔️✔️
Eye gaze✔️
Hands / Gestures✔️✔️
Motion controllers✔️
Perception and spatial features HoloLens (1st gen)HoloLens 2 Immersive headsets
World coordinates✔️✔️✔️
Spatial sound✔️✔️✔️
Spatial mapping✔️✔️

The basic interaction model for HoloLens is gaze, gesture, and voice, sometimes referred to as GGV. Windows Mixed Reality immersive headsets also use gaze and voice, but swap motion controllers for gestures.

All Windows-based mixed reality devices benefit from the input ecosystem available to Windows, including mouse, keyboard, gamepads, and more. With HoloLens, hardware accessories are connected via Bluetooth. With immersive headsets, accessories connect to the host PC via Bluetooth, USB, and other supported protocols.

The environmental understanding features like coordinates, spatial sound, and spatial mapping provide the necessary capabilities for mixing reality. Spatial mapping is unique to HoloLens, and enables holograms to interact with both the user and the physical world around them. Coordinate systems allow the user's movement to affect movement in the digital world.

Holograms are made of light and sound, which rely on holographic rendering. Understanding the experience of placement and persistence, as demonstrated in the Windows Mixed Reality home (sometimes called the "shell") is a great way ground yourself in the user experience.

Tools for developing for mixed reality

The tools you use will depend on the style of app you want to build.

  • Apps with a 2D view leverage tools for building Universal Windows Platform apps suited for environments like Windows Phone, PC, and tablets. These apps are experienced as 2D projections placed in the Windows Mixed Reality home, and can work across multiple device types (including phone and PC).
  • Immersive and holographic apps need tools designed to take advantage of the Windows Mixed Reality APIs. We recommend using Unity to build mixed reality apps. Developers interested in building their own engine can use DirectX and other Windows APIs.

Regardless of the type of app you're building, these tools will facilitate your app development experience:

See also