HoloLens Photo/Video Camera in Unreal
The HoloLens has a Photo/Video (PV) Camera on the visor that can be used for both Mixed Reality Capture (MRC) and locating objects in Unreal world space from pixel coordinates in the camera frame.
The PV Camera isn't supported with Holographic Remoting, but it's possible to use a webcam attached to your PC to simulate the HoloLens PV Camera functionality.
Render from the PV Camera for MRC
This requires Unreal Engine 4.25 or newer.
The system and custom MRC recorders create mixed reality captures by combining the PV Camera with holograms rendered by the app.
By default, mixed reality capture uses the right eye's holographic output. If an immersive app chooses to render from the PV Camera, then that will be used instead. Rendering from the PV Camera improves the mapping between the real world and the holograms in the MRC video.
To opt in to rendering from the PV Camera:
- Call SetEnabledMixedRealityCamera and ResizeMixedRealityCamera
- Use the Size X and Size Y values to set the video dimensions.
Unreal will then handle requests from MRC to render from the PV Camera's perspective.
Only when Mixed Reality Capture is triggered will the app be asked to render from the photo/video camera's perspective.
Using the PV Camera
The webcam texture can be retrieved in the game at runtime, but it needs to be enabled in the editor's Edit > Project Settings:
- Go to Platforms > HoloLens > Capabilities and check Webcam.
- Use the StartCameraCapture function to use the webcam at runtime and the StopCameraCapture function to stop recording.
Rendering an image
To render the camera image:
- Create a dynamic material instance based on a material in the project, which is named PVCamMat in the screenshot below.
- Set the dynamic material instance to a Material Instance Dynamic Object Reference variable.
- Set the material of the object in the scene that will render the camera feed to this new dynamic material instance.
- Start a timer that will be used to bind the camera image to the material.
- Create a new function for this timer, in this case MaterialTimer, and call GetARCameraImage to get the texture from the webcam.
- If the texture is valid, set a texture parameter in the shader to the image. Otherwise, start the material timer again.
- Make sure the material has a parameter that matches the name in SetTextureParameterValue that's bound to a color entry. Without the parameter, the camera image can't be displayed properly.
Next Development Checkpoint
If you're following the Unreal development journey we've laid out, you're in the midst of exploring the Mixed Reality platform capabilities and APIs. From here, you can continue to the next topic:
Or jump directly to deploying your app on a device or emulator:
You can always go back to the Unreal development checkpoints at any time.