Training
Module
Render a 3D model with Azure Remote Rendering - Training
Use Azure Remote Rendering to render a 3D model in a Unity project. You can deploy the model to HoloLens 2 or use the power of mixed reality with the MRTK.
This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
Holographic rendering enables your application to draw a hologram at a precise location in the world around the user, whether it's precisely placed in the physical world or within a virtual realm you've created. Holograms are objects made of sound and light. Rendering enables your application to add the light.
Feature | HoloLens (first gen) | HoloLens 2 | Immersive headsets |
Rendering | ✔️ | ✔️ | ✔️ |
Key to holographic rendering is knowing what kind of device is being used. Devices with see-through displays, such as HoloLens, add light to the world. Black pixels are fully transparent, while brighter pixels are increasingly opaque. Because the light from the displays is added to the light from the real world, white pixels are translucent.
While stereoscopic rendering provides one depth cue for your holograms, adding grounding effects can help users see more easily what surface a hologram is near. One grounding technique is to add a glow around a hologram on the nearby surface, and then render a shadow against this glow. In this way, your shadow appears to subtract light from the environment. Spatial sound is another important depth cue, letting users reason about the distance and relative location of a hologram.
Devices with opaque displays, like Windows Mixed Reality immersive headsets, block out the world. Black pixels are solid black, and any other color appears as that color to the user. Your application is responsible for rendering everything the user sees. This makes it even more important to maintain a constant refresh rate so that users have a comfortable experience.
Mixed reality headsets (both HoloLens and immersive headsets) continually track the position and orientation of the user's head relative to their surroundings. As your application begins preparing its next frame, the system predicts where the user's head will be in the future at the exact moment that the frame shows up on the displays. Based on this prediction, the system calculates the view and the projection transforms to use for that frame. Your application must use these transforms to produce correct results. If system-supplied transforms aren't used, the resulting image won't align with the real world, leading to user discomfort.
Note
To accurately predict when a new frame will reach the displays, the system is constantly measuring the effective end-to-end latency of your application's rendering pipeline. While the system adjusts to the length of your rendering pipeline, you can improve hologram stability by keeping that pipeline as short as possible.
Applications that use advanced techniques to augment the system prediction can override the system view and projection transforms. These applications must still use system-supplied transforms as a basis for their custom transforms to produce meaningful results.
When rendering a frame, the system specifies the back-buffer viewport in which your application should draw. This viewport is often smaller than the full size of the frame buffer. Whatever the viewport size, once the frame is rendered by the application, the system upscales the image to fill the entirety of the displays.
For applications that find themselves unable to render at the required refresh rate, system rendering parameters can be configured to reduce memory pressure and rendering cost at the cost of increased pixel aliasing. The back buffer format can also be changed, which for some apps can help to improve memory bandwidth and pixel throughput.
The rendering frustum, resolution, and framerate in which your app is asked to render might also change from frame to frame, and might differ across the left and right eye. For example, when mixed reality capture (MRC) is active and the photo/video camera view configuration isn't opted-into, one eye might be rendered with a larger FOV or resolution.
For any given frame, your app must render using the view transform, projection transform, and viewport resolution provided by the system. Additionally, your application must never assume that any rendering or view parameter remains fixed from frame-to-frame. Engines like Unity handle all these transforms for you in their own camera objects so that the physical movement of your users and the state of the system is always respected. If your application allows for virtual movement of the user through the world (e.g. using the thumbstick on a gamepad), you can add a parent rig object above the camera that moves it around. This causes the camera to reflect both the user's virtual and physical motion. If your application modifies the view transform, projection transform, or viewport dimension provided by the system, it must inform the system by calling the appropriate override API.
To enhance the stability of your holographic rendering, your app should provide to Windows each frame the depth buffer it used for rendering. If your app does provide a depth buffer, it should have coherent depth values, with depth expressed in meters from the camera. This enables the system to use your per-pixel depth data to better stabilize content if the user's head ends up slightly offset from the predicted location. If you aren't able to provide your depth buffer, you can provide a focus point and normal, defining a plane that cuts through most of your content. If both the depth buffer and a focus plane are provided, the system might use both. In particular, it's helpful to provide both the depth buffer and a focus point that includes a velocity vector when your application displays holograms that are in motion.
Refer to Rendering in DirectX article for low-level details about this topic.
Windows Mixed Reality introduces the concept of a holographic camera. Holographic cameras are similar to the traditional camera found in 3D graphics texts; they define both the extrinsic (position and orientation) and intrinsic camera properties. (For example, field-of-view is used to view a virtual 3D scene.) Unlike traditional 3D cameras, the application isn't in control of the position, orientation, and intrinsic properties of the camera. Rather, the position and orientation of the holographic camera is implicitly controlled by the user's movement. The user's movement is relayed to the application on a frame-by-frame basis via a view transform. Likewise, the camera's intrinsic properties are defined by the device's calibrated optics and relayed frame-by-frame via the projection transform.
In general, your application will render for a single stereo camera. A robust rendering loop will support multiple cameras, and will support both mono and stereo cameras. For example, the system might ask your application to render from an alternate perspective when the user activates a feature like mixed reality capture (MRC), depending on the headset shape. Applications that can support multiple cameras get them by opting-in to the kind of cameras they can support.
When rendering medical MRIs or engineering volumes in 3D, volume rendering techniques are often used. These techniques can be interesting in mixed reality, where users can naturally view such a volume from key angles, simply by moving their head.
Training
Module
Render a 3D model with Azure Remote Rendering - Training
Use Azure Remote Rendering to render a 3D model in a Unity project. You can deploy the model to HoloLens 2 or use the power of mixed reality with the MRTK.