HoloLens Research mode
This feature was added as part of the Windows 10 RS4 update for HoloLens, and is not available on earlier releases. You can test it now by installing the HoloLens RS4 Preview. Make sure your application is targeting a version of the Windows SDK greater than or equal to 10.0.17125 on HoloLens. You can find the latest Windows SDK here.
Research mode is a new capability of HoloLens that provides application access to the key sensors on the device. These include:
- The four environment tracking cameras used by the system for map building and head tracking.
- Two versions of the depth camera data – one for high-frequency (30 FPS) near-depth sensing, commonly used in hand tracking, and the other for lower-frequency (1 FPS) far-depth sensing, currently used by Spatial Mapping,
- Two versions of an IR-reflectivity stream, used by the HoloLens to compute depth, but valuable in its own right as these images are illuminated from the HoloLens and reasonably unaffected by ambient light.
A mixed reality capture of a test application that displays the eight sensor streams available in Research mode
Before using Research mode
Research mode is well named: it is intended for academic and industrial researchers trying out new ideas in the fields of Computer Vision and Robotics. Research mode is not intended for applications that will be deployed across an enterprise or made available in the Microsoft Store. The reason for this is that Research mode lowers the security of your device and consumes significantly more battery power than normal operation. Microsoft is not committing to supporting this mode on any future devices. Thus, we recommend you use it to develop and test new ideas; however, you will not be able to widely deploy applications that use Research mode or have any assurance that it will continue to work on future hardware.
Enabling Research mode
Research mode is a sub-mode of developer mode. You first need to enable developer mode in the Settings app (Settings > Update & Security > For developers):
- Set "Use developer features" to On
- Set "Enable Device Portal" to On
Then using a web browser that is connected to the same Wi-Fi network as your HoloLens, navigate to the IP address of your HoloLens (obtained through Settings > Network & Internet > Wi-Fi > Hardware properties). This is the Device Portal, and you will find a "Research mode" page in the "System" section of the portal:
Research mode in the HoloLens Device Portal
After selecting Allow access to sensor streams, you will need to reboot HoloLens. You can do this from the Device Portal, under the "Power" menu item at the top of the page.
Once your device has rebooted, applications that have been loaded through Device Portal should be able to access Research mode streams.
Using sensor data in your apps
Applications can access sensor stream data by opening Media Foundation streams in exactly the same way they access the photo/video camera stream.
All APIs that work for HoloLens development are also available when in Research mode. In particular, the application can know precisely where HoloLens is in 6DoF space at each sensor frame capture time.
Sample applications showing how you access the various Research mode streams, how to use the intrinsics and extrinsics, and how to record streams are available in the HoloLensForCV GitHub repo.
See the issue tracker in the HoloLensForCV repository.