HoloLens Photo/Video Camera in Unreal

The HoloLens has a Photo/Video (PV) Camera on the visor that can be used for both Mixed Reality Capture (MRC) and locating objects in Unreal world space from pixel coordinates in the camera frame.

Important

The PV Camera isn't supported with Holographic Remoting, but it's possible to use a webcam attached to your PC to simulate the HoloLens PV Camera functionality.

Render from the PV Camera for MRC

Note

This requires Unreal Engine 4.25 or newer.

The system and custom MRC recorders create mixed reality captures by combining the PV Camera with holograms rendered by the app.

By default, mixed reality capture uses the right eye's holographic output. If an immersive app chooses to render from the PV Camera, then that will be used instead. Rendering from the PV Camera improves the mapping between the real world and the holograms in the MRC video.

To opt in to rendering from the PV Camera:

  1. Call SetEnabledMixedRealityCamera and ResizeMixedRealityCamera
    • Use the Size X and Size Y values to set the video dimensions.

Camera 3rd

Unreal will then handle requests from MRC to render from the PV Camera's perspective.

Note

Only when Mixed Reality Capture is triggered will the app be asked to render from the photo/video camera's perspective.

Using the PV Camera

The webcam texture can be retrieved in the game at runtime, but it needs to be enabled in the editor's Edit > Project Settings:

  1. Go to Platforms > HoloLens > Capabilities and check Webcam.
    • Use the StartCameraCapture function to use the webcam at runtime and the StopCameraCapture function to stop recording.

Camera Start Stop

Rendering an image

To render the camera image:

  1. Create a dynamic material instance based on a material in the project, which is named PVCamMat in the screenshot below.
  2. Set the dynamic material instance to a Material Instance Dynamic Object Reference variable.
  3. Set the material of the object in the scene that will render the camera feed to this new dynamic material instance.
    • Start a timer that will be used to bind the camera image to the material.

Camera Render

  1. Create a new function for this timer, in this case MaterialTimer, and call GetARCameraImage to get the texture from the webcam.
  2. If the texture is valid, set a texture parameter in the shader to the image. Otherwise, start the material timer again.

Camera Texture from webcam

  1. Make sure the material has a parameter that matches the name in SetTextureParameterValue that's bound to a color entry. Without the parameter, the camera image can't be displayed properly.

Camera Texture

Next Development Checkpoint

If you're following the Unreal development journey we've laid out, you're in the midst of exploring the Mixed Reality platform capabilities and APIs. From here, you can continue to the next topic:

Or jump directly to deploying your app on a device or emulator:

You can always go back to the Unreal development checkpoints at any time.

See also