Unreal Development Overview
Getting started with mixed reality applications is a big task. New concepts, platforms, and cutting edge hardware can seem like barriers. However, if you're an Unreal developer you're in luck. Unreal Engine 5 has full support for Windows Mixed Reality (VR) and HoloLens 2 (AR) devices.
The latest Unreal Engine 4.27+ releases include:
- Windows Mixed Reality (WMR) plugin replaced by OpenXR and Microsoft OpenXR plugin (Marketplace or GitHub)
- OpenXR offers feature parity with WMR
- Unified API for cross platform development
- OpenXR available for application creation in-engine — full feature set available with Microsoft OpenXR plugin
- Microsoft OpenXR plugin updates release when ready instead of in-engine updates
- Use the Graphics Tools Plugin for Unreal to improve visual fidelity of Mixed Reality applications while maintaining performance
- HoloLensAR plugin is removed in Unreal 5.0
- Support for World Locking Tools (5.0+)
- Support for Azure Object Anchors (4.27+)
See the Unreal Engine 5.0 release notes for more.
If you're new to Unreal development, don't jump in blind. Explore the Unreal resources and look for assets in the Unreal marketplace. You can also find support in the mixed reality forums and Unreal documentation. These resources are your links to the community of builders and problem solvers in today's mixed reality market.
Important
Take a look at our porting guide if you have an existing Unreal project that you want to bring over to immersive headsets such as the Reverb G2.
Development checkpoints
Use the following checkpoints to bring your Unreal games and applications into the world of mixed reality. If you haven't explored the Designing Holograms sample application, we recommend downloading it to familiarize yourself with the basics of Mixed Reality UX.
1. Getting started
First, you'll need to install the tools for HoloLens 2 development. Next, go through our tutorial series to get a basic understanding of the Mixed Reality Toolkit, a properly configured development environment for mixed reality apps, and a working MRTK project in Unreal. As of Unreal 4.26, you also have the option of developing an OpenXR app for HoloLens 2.
| Checkpoint | Outcome |
|---|---|
| Install the latest tools | Download and install the latest version of Unreal Engine and setup your project for mixed reality |
| Set up your project | Get the latest version of Unreal Engine and MRTK |
| Creating your first HoloLens Unreal application | Start off your Unreal and HoloLens development journey by building a basic Mixed Reality application |
| HoloLens 2 tutorial series | Get set up for mixed reality development in Unreal, build your first app with MRTK, and deploy your app to HoloLens 2 |
| Get started with OpenXR in Unreal | Install and enable the following plugin from the Unreal Engine Marketplace:Ensure the Microsoft Windows Mixed Reality plugin is disabled. The full list of currently supported features in OpenXR is below. |
2. Core building blocks
There are a number of key mixed reality features that our tutorial series doesn't cover. These building blocks are available as standalone features and through the Mixed Reality Toolkit. You might not need all of them at once, but we recommend exploring early on. After diving into the core building blocks listed below, you'll have a toolbox full of features you can integrate into your Mixed Reality projects.
The Mixed Reality Toolkit for Unreal is a set of plugins designed to speed up your development in Unreal. Each plugin includes components, samples, and documentation for setting up immersive experiences.
UX Tools for Unreal is the first plugin to be released and is currently only supported on HoloLens 2. The plugin includes C++ code, Blueprints, and example assets of common UX features for input simulation, hand interactions, surface magnetism and more.
Graphics Tools for Unreal is a UE game plugin with code, blueprints and example assets created to help improve the visual fidelity of Mixed Reality applications while staying within performance budgets.
| Feature | Capabilities |
|---|---|
| Input simulation | Use mouse and keyboard input in the editor to simulate a Head-Mounted Display (HMD) and hand tracking features |
| Hand interaction | Interact with UX components using articulated hand inputs |
| Pressable button | Drive the state of button blueprints through low level components |
| Manipulator | Move and rotate an actor with one or two-hand manipulation input |
| Follow behavior | Keep an actor or component within sight of another component |
Note
You can dive into the UX Tools for Unreal GitHub repository for more details.
3. Advanced features
Other key features that play a role in mixed reality applications are available without any extra packages or setup. These features can be added to Unreal projects with or without MRTK installed. After diving into these more advanced capabilities, you'll be able to build more complex Mixed Reality apps.
| Feature | Capabilities |
|---|---|
| HoloLens camera | Capture Mixed Reality and real-world visual content from your app running on a HoloLens device |
| QR codes | Render QR codes as holograms using a coordinate system at each code's real-world position |
| WinRT | Create a separate binary with WinRT code that can be consumed by Unreal’s build system |
4. Streaming and deploying to a device
If you want to test out your application on a HoloLens device while still in development, you can stream it directly from your PC by using either the Unreal editor or a packaged Windows executable.
If this is your first time deploying an Unreal app to HoloLens 2, you'll need to download supporting files from the Epic Launcher. Once you have those files installed, you're ready to deploy from either the Unreal editor or the Device Portal.
5. Adding services
At this point in your development journey, you might be looking to add services or for a helping hand with commercial deployment. Integrating Azure Cloud Services can level up your projects in a major way. We've compiled a few starting points for you to explore and expand your Mixed Reality knowledge.
Azure Spatial Anchors (ASA) is a cross-platform service that allows you to build spatially aware mixed reality applications. With Azure Spatial Anchors, you can map, persist, and share holographic content across multiple devices, at real-world scale.
6. Low-code alternatives
Microsoft Dynamics 365 Guides is a mixed-reality application for Microsoft HoloLens that lets operators learn during the flow of work by providing holographic instructions when and where they are needed. These instruction cards are visually tethered to the place where the work must be done, and can include images, videos, and 3D holographic models.
What's next?
A developer's job is never done, especially when learning a new tool or SDK. The following sections can take you into areas beyond the beginner level material you've already completed, along with helpful resources if you get stuck. Note that these topics and resources are not in any sequential order, so feel free to jump around and explore!
Debugging
If you're looking to debug the application while it's running on device with Visual Studio, follow these instructions.
Performance
Developing for mixed reality comes with performance checkpoints that depend on the platform. A HoloLens 2 app must run at 60 frames per second for holograms to appear stable and responsive. Luckily, we have performance recommendations for upgrading performance in your Unreal applications.
Supported Features
| HoloLens 2 Feature | Earliest Supported Unreal Engine Version | Supported in OpenXR (4.26+) |
|---|---|---|
| ARM64 support | 4.23 | ✔️ |
| Streaming from a PC | 4.23 | ✔️ |
| Spatial mapping | 4.23 | ✔️ |
| Hand and joint tracking | 4.23 | ✔️ |
| Eye tracking | 4.23 | ✔️ |
| Voice input | 4.23 | ✔️ |
| Spatial anchors | 4.23 | ✔️ |
| Camera access | 4.23 | ✔️ |
| QR codes | 4.23 | ✔️ |
| Spatial audio | 4.23 | ✔️ |
| Planar LSR over streaming | 4.24 | |
| Sample apps | 4.24 | ✔️ |
| Mobile multi-View: Performance hits 60 fps | 4.25 | ✔️ |
| Streaming from a packaged desktop app | 4.25.1 | ✔️ |
| Azure Spatial Anchors for HoloLens 2 | 4.25 | ✔️ |
| Mixed Reality UX Tools support | 4.25 | ✔️ |
| Developer docs & tutorials | 4.25 | ✔️ |
| System keyboard | 4.26 | ✔️ |
| HoloLens Media Player plugin | 4.26 | ✔️ |
| Azure Spatial Anchors for iOS and Android | 4.26 | |
| Microsoft OpenXR plugin with Microsoft vendor-specific OpenXR extensions | 4.26 | ✔️ |
| Streaming from Azure to HoloLens 2 | 4.26 | ✔️ |
| Windows App Certification Kit compliance for packaged apps | 4.26 | ✔️ |
| HP Reverb G2 controller support | 4.26 | ✔️ |
See also
Feedback
Coming soon: Throughout 2024 we will be phasing out GitHub Issues as the feedback mechanism for content and replacing it with a new feedback system. For more information see: https://aka.ms/ContentUserFeedback.
Submit and view feedback for