Unreal Development Overview
Getting started with mixed reality applications is a big task. New concepts, platforms, and cutting edge hardware can seem like barriers. However, if you're an Unreal developer you're in luck. Unreal Engine 4 has full support for Windows Mixed Reality (VR) and HoloLens 2 (AR) devices.
The latest Unreal Engine 4.26 release includes:
- Support for new cross-device XR APIs
- OpenXR support + Microsoft OpenXR plugin
- HoloLens Media Player plugin for performant video playback
- Azure Spatial Anchors for iOS and Android (beta)
- Support for reverse connections when remoting, enabling streaming from Azure to HoloLens 2
- HoloLens 2 system keyboard access
- Windows App Certification Kit compliance for packaged apps
- Unreal Insights support for HoloLens 2
- HP Reverb G2 controller support
See the Unreal Engine 4.26 release notes for more.
If you're new to Unreal development, don't jump in blind. Explore the Unreal tutorial series and look for assets in the Unreal marketplace. You can also find support in the mixed reality forums. These resources are your links to the community of builders and problem solvers in today's mixed reality market.
Take a look at our porting guide if you have an existing Unreal project that you want to bring over to immersive headsets such as the Reverb G2.
Use the following checkpoints to bring your Unreal games and applications into the world of mixed reality. If you haven't explored the Designing Holograms sample application, we recommend downloading it to familiarize yourself with the basics of Mixed Reality UX.
1. Getting started
First, you'll need to install the tools for HoloLens 2 development. Next, go through our tutorial series to get a basic understanding of the Mixed Reality Toolkit, a properly configured development environment for mixed reality apps, and a working MRTK project in Unreal. As of Unreal 4.26, you also have the option of developing an OpenXR app for HoloLens 2.
|Install the latest tools||Download and install the latest version of Unreal Engine and setup your project for mixed reality|
|Creating your first HoloLens Unreal application||Start off your Unreal and HoloLens development journey by building a basic Mixed Reality application|
|HoloLens 2 tutorial series||Get set up for mixed reality development in Unreal, build your first app with MRTK, and deploy your app to HoloLens 2|
|(Optional) Get started with OpenXR in Unreal||If you're looking to build an OpenXR app in Unreal, you should disable the following engine plugin:
Download and enable the following plugin in your project from GitHub:
The full list of currently supported features in OpenXR is below.
2. Core building blocks
There are a number of key mixed reality features that our tutorial series doesn't cover. These building blocks are available as standalone features and through the Mixed Reality Toolkit. You might not need all of them at once, but we recommend exploring early on. After diving into the core building blocks listed below, you'll have a toolbox full of features you can integrate into your Mixed Reality projects.
The Mixed Reality Toolkit for Unreal is a set of plugins designed to speed up your development in Unreal. Each plugin includes components, samples, and documentation for setting up immersive experiences.
UX Tools for Unreal is the first plugin to be released and is currently only supported on HoloLens 2. The plugin includes C++ code, Blueprints, and example assets of common UX features for input simulation, hand interactions, surface magnetism and more.
Graphics Tools for Unreal is a UE game plugin with code, blueprints and example assets created to help improve the visual fidelity of Mixed Reality applications while staying within performance budgets.
|Input simulation||Use mouse and keyboard input in the editor to simulate a Head-Mounted Display (HMD) and hand tracking features|
|Hand interaction||Interact with UX components using articulated hand inputs|
|Pressable button||Drive the state of button blueprints through low level components|
|Manipulator||Move and rotate an actor with one or two-hand manipulation input|
|Follow behavior||Keep an actor or component within sight of another component|
You can dive into the UX Tools for Unreal GitHub repository for more details.
3. Advanced features
Other key features that play a role in mixed reality applications are available without any extra packages or setup. These features can be added to Unreal projects with or without MRTK installed. After diving into these more advanced capabilities, you'll be able to build more complex Mixed Reality apps.
|HoloLens camera||Capture Mixed Reality and real-world visual content from your app running on a HoloLens device|
|QR codes||Render QR codes as holograms using a coordinate system at each code's real-world position|
|WinRT||Create a separate binary with WinRT code that can be consumed by Unreal’s build system|
4. Streaming and deploying to a device
If you want to test out your application on a HoloLens device while still in development, you can stream it directly from your PC by using either the Unreal editor or a packaged Windows executable.
If this is your first time deploying an Unreal app to HoloLens 2, you'll need to download supporting files from the Epic Launcher. Once you have those files installed, you're ready to deploy from either the Unreal editor or the Device Portal.
5. Adding services
At this point in your development journey, you might be looking to add services or for a helping hand with commercial deployment. Integrating Azure Cloud Services and Dynamics 365 features can level up your projects in a major way. We've compiled a few starting points for you to explore and expand your Mixed Reality knowledge.
Azure Spatial Anchors (ASA) is a cross-platform service that allows you to build spatially aware mixed reality applications. With Azure Spatial Anchors, you can map, persist, and share holographic content across multiple devices, at real-world scale.
A developer's job is never done, especially when learning a new tool or SDK. The following sections can take you into areas beyond the beginner level material you've already completed, along with helpful resources if you get stuck. Note that these topics and resources are not in any sequential order, so feel free to jump around and explore!
If you're looking to debug the application while it's running on device with Visual Studio, follow these instructions.
Developing for mixed reality comes with performance checkpoints that depend on the platform. A HoloLens 2 app must run at 60 frames per second for holograms to appear stable and responsive. Luckily, we have performance recommendations for upgrading performance in your Unreal applications.
|HoloLens 2 Feature||Earliest Supported Unreal Engine Version||Supported in OpenXR (4.26)|
|Streaming from a PC||4.23||✔️|
|Hand and joint tracking||4.23||✔️|
|Spectator Screen support for streaming||4.24|
|Planar LSR over streaming||4.24|
|Mobile multi-View: Performance hits 60 fps||4.25||✔️|
|3rd camera render||4.25|
|Streaming from a packaged desktop app||4.25.1||✔️|
|Azure Spatial Anchors for HoloLens 2 (beta)||4.25|
|Mixed Reality UX Tools support||4.25||✔️|
|Developer docs & tutorials||4.25||✔️|
|HoloLens Media Player plugin||4.26||✔️|
|Azure Spatial Anchors for iOS and Android (beta)||4.26|
|Microsoft OpenXR plugin with Microsoft vendor-specific OpenXR extensions||4.26||✔️|
|Streaming from Azure to HoloLens 2||4.26||✔️|
|Windows App Certification Kit compliance for packaged apps||4.26||✔️|
|HP Reverb G2 controller support||4.26||✔️|