This document describes the top factors impacting the quality of mixed reality apps. For each factor the following information is provided
Overview – a brief description of the quality factor and why it is important.
Device impact - which type of Window Mixed Reality device is impacted.
Quality criteria – how to evaluate the quality factor.
How to measure – methods to measure (or experience) the issue.
Recommendations – summary of approaches to provide a better user experience.
Resources – relevant developer and design resources that are useful to create better app experiences.
Frame rate is the first pillar of hologram stability and user comfort. Frame rate below the recommended targets can cause holograms to appear jittery, negatively impacting the believability of the experience and potentially causing eye fatigue. The target frame rate for your experience on Windows Mixed Reality immersive headsets will be either 60Hz or 90Hz depending on which Windows Mixed Reality Compatible PCs you wish to support. For HoloLens the target frame rate is 60Hz.
The app consistently meets frames per second (FPS) goal for target device: 60fps on HoloLens; 90fps on Ultra PCs; and 60fps on mainstream PCs.
The app has intermittent frame drops not impeding the core experience; or FPS is consistently lower than desired goal but doesn’t impede the app experience.
The app is experiencing a drop in frame rate on average every ten seconds or less.
How to measure
A real-time frame rate graph is provided through by the Windows Device Portal under "System Performance".
For development debugging, add a frame rate diagnostic counter into the app. See Resources for a sample counter.
Frame rate drops can be experienced in device while the app is running by moving your head from side to side. If the hologram shows unexpected jittery movement, then low frame rate or the stability plane is likely the cause.
Add a frame rate counter at the beginning of the development work.
Changes that incur a drop in frame rate should be evaluated and appropriately resolved as a performance bug.
Stable holograms will increase the usability and believability of your app, and create a more comfortable viewing experience for the user. The quality of hologram stability is a result of good app development and the device's ability to understand (track) its environment. While frame rate is the first pillar of stability, other factors can impact stability including:
Secondary content exhibits unexpected movement; or unexpected movement does not impede overall app experience.
Primary content in frame exhibits unexpected movement.
How to measure
While wearing the device and viewing the experience:
Move your head from side to side, if the holograms show unexpected movement then low frame rate or improper alignment of the stability plane to the focal plane is the likely cause.
Move around the holograms and environment, look for behaviors such as swim and jumpiness. This type of motion is likely caused by the device not tracking the environment, or the distance to the spatial anchor.
If large or multiple holograms are in the frame, observe hologram behavior at various depths while moving your head position from side to side, if shakiness appears this is likely caused by the stabilization plane.
Add an frame rate counter at the beginning of the development work.
Use the stabilization plane.
Always render anchored holograms within 3 meters of their anchor.
Make sure your environment is setup for proper tracking.
Design your experience to avoid holograms at various focal depth levels within the frame.
Misalignments of holograms with physical objects (if intended to be placed in relation to one another) is a clear indication of the non-union of holograms and real-world. Accuracy of the placement should be relative to the needs of the scenario; for example, general surface placement can use the spatial map, but more accurate placement will require some use of markers and calibration.
Holograms align to the surface typically in the centimeters to inches range. If more accuracy is required, the app should provide an efficient means for collaboration within the desired app spec.
The holograms appear unaligned with the physical target object by either breaking the surface plane or appearing to float away from the surface. If accuracy is required, Holograms should meet the proximity spec of the scenario.
How to measure
Holograms that are placed on spatial map should not appear to dramatically float above or below the surface.
Holograms that require accurate placement should have some form of marker and calibration system that is accurate to the scenario's requirement.
Spatial map is useful for placing objects on surfaces when precision isn’t required.
For the best precision, use markers or posters to set the holograms and an Xbox controller (or some manual alignment mechanism) for final calibration.
Consider breaking extra-large holograms into logical parts and aligning each part to the surface.
Improperly set interpupilary distance (IPD) can also effect hologram alignment. Always configure HoloLens to the user's IPD.
App developers control where users' eyes converge by placing content and holograms at various depths. Users wearing HoloLens will always accommodate to 2.0m to maintain a clear image because HoloLens displays are fixed at an optical distance approximately 2.0m away from the user. Improper content depth can lead to visual discomfort or fatigue.
Regardless of viewing zone of comfort issues, demands for the user to switch frequently or quickly between near and far focal objects (including holograms and real-world content) can lead to oculomotor fatigue, and general discomfort.
In Windows Mixed Reality, the audio engine provides the aural component of the mixed reality experience by simulating 3D sound using direction, distance, and environmental simulations. Using spatial sound in an application allows developers to convincingly place sounds in a 3 dimensional space (sphere) all around the user. Those sounds will then seem as if they were coming from real physical objects or the mixed reality holograms in the user's surroundings. Spatial sound is a powerful tool for immersion, accessibility, and UX design in mixed reality applications.
Well-designed user experiences can create and maintain useful context of the virtual environment that extends around the users. Mitigating the effect of the FOV boundaries involves a thoughtful design of content scale and context, use of spatial audio, guidance systems, and the user's position. If done right, the user will feel less impaired by the FOV boundaries while having a comfortable app experience.
User never loses context and viewing is comfortable. Context assistance is provided for large objects. Discoverability and viewing guidance is provided for objects outside the frame. In general, motion design and scale of the holograms are appropriate for a comfortable viewing experience.
User never loses context, but extra neck motion may be required in limited situations. In limited situations scale causes holograms to break either the vertical or horizontal frame causing some neck motion to view holograms.
User likely to lose context and/or consistent neck motion is required to view holograms. No context guidance for large holographic objects, moving objects easy to lose outside the frame with no discoverability guidance, or tall holograms requires regular neck motion to view.
How to measure
Context for a (large) hologram is lost or not understood due to being clipped at the boundaries.
Location of holograms are hard to find due to the lack of attention directors or content that rapidly moves in and out of the holographic frame.
Scenario requires regular and repetitive up and down head motion to fully see a hologram resulting in neck fatigue.
Start the experience with small objects that fit the FOV, then transition with visual cues to larger versions.
Use spatial audio and attention directors to help the user find content that is outside the FOV.
As much as possible, avoid holograms that vertically clip the FOV.
Provide the user with in-app guidance for best viewing location.
Holograms should react to the user position in roughly the same ways that "real" objects do. A notable design consideration is UI elements that can't necessarily assume a user's position is stationary and adapt to the user's motion. Designing an app that correctly adapts to user position will create a more believable experience and make it easier to use.
Content and UI adapt to user positions allowing user to naturally interact with content within the scope of expected user movement.
UI adapts to the user position, but may impede the view of key content requiring the user to adjust their position.
UI elements are lost or locked during movement causing user to unnaturally return to (or find) controls.
UI elements limit the view of primary content.
UI movement is not optimized for viewing distance and momentum particularly with tag-along UI elements.
How to measure
All measurements should be done within a reasonable scope of the scenario. While user movement will vary, don’t try to trick the app with extreme user movement.
For UI elements, relevant controls should be available regardless of user movement. For example, if the user is viewing and walking around a 3D map with zoom, the zoom control should be readily available to the user regardless of location.
The user is the camera and they control the movement. Let them drive.
Consider billboarding for text and menuing systems that would otherwise be world-locked or obscured if a user were to move around.
Use tag-along for content that needs to follow the user while still allowing the user to see what is in front of them.
Input interaction clarity is critical to an app's usability and includes input consistency, approachability, discoverability of interaction methods. User should be able to use platform-wide common interactions without relearning. If the app has custom input, it should be clearly communicated and demonstrated.
Input interaction methods are consistent with Windows Mixed Reality provided guidance. Any custom input should not be redundant with standard input (rather use standard interaction) and must be clearly communicated and demonstrated to the user.
Similar to best, but custom inputs are redundant with standard input methods. User can still achieve the goal and progress through the app experience.
Difficult to understand input method or button mapping. Input is heavily customized, does not support standard input, no instructions, or likely to cause fatigue and comfort issues.
A button has long been a metaphor used for triggering an event in the 2D abstract world. In the three-dimensional mixed reality world, we don’t have to be confined to this world of abstraction anymore. Anything can be an Interactable object that triggers an event. An interactable object can be represented as anything from a coffee cup on the table to a balloon floating in the air. Regardless of the form, interactable objects should be clearly recognizable by the user through visual and audio cues.
Regardless of form, interactable objects are recognizable through visual and audio cues across three states: idle, targeted, and selected. "See it, say it" is clear and consistently used throughout the experience. Objects are scaled and distributed to allow for error free targeting.
User can recognize object as interactable through audio or visual feedback, and can target and activate the object.
Given no visual or audio cues, user cannot recognize an interactable object. Interactions are error prone due to object scale or distance between objects.
How to measure
Interactable objects are recognizable as 'interactable'; including buttons, menus, and app specific content. As a rule of thumb there should be a visual and audio cue when targeting interactable objects.
Use visual and audio feedback for interactions.
Visual feedback should be differentiated for each input state (idle, targeted, selected)
Interactable objects should be scaled and placed for error free targeting.
Grouped interactable objects (such as a menu bar or list) should have proper spacing for targeting.
Buttons and menus that support voice command should provide text labels for the command keyword ("See it, say it")
Apps that require spatial mapping data rely on the device to automatically collect this data over time and across sessions as the user explores their environment with the device active. The completeness and quality of this data depends on a number of factors including the amount of exploration the user has done, how much time has passed since the exploration and whether objects such as furniture and doors have moved since the device scanned the area. Many apps will analyze the spatial mapping data at the start of the experience to judge whether the user should perform additional steps to improve the completeness and quality of the spatial map. If the user is required to scan the environment, clear guidance should be provided during the scanning experience.
Visualization of the spatial mesh tell users scanning is in progress. User clearly knows what to do and when the scan starts and stops.
Visualization of the spatial mesh is provided, but the user may not clearly know what to do and no progress information is provided.
No visualization of mesh. No guidance information provided to the user regarding where to look, or when the scan starts/stops.
How to measure
During a required room scan, visual and audio guidance is provided indicating where to look, and when to start and stop scanning.
Indicate how much of the total volume in the users vicinity needs to be part of the experience.
Communicate when the scan starts and stops such as a progress indicator.
Use a visualization of the mesh during the scan.
Provide visual and audio cues to encourage the user to look and move around the room.
Inform the user where to go to improve the data. In many cases, it may be best to tell the user what they need to do (e.g. look at the ceiling, look behind furniture), in order to get the necessary scan quality.
In a mixed reality app, content may be outside the field of view or occluded by real-world objects. A well designed app will make it easier for the user to find non-visible content. Directional indicators alert a user to important content and provide guidance to the content relative to the user's position. Guidance to non-visible content can take the form of sound emitters, directional arrows, or direct visual cues.
Visual and audio cues directly guide the user to relevant content outside the field of view.
An arrow or some indicator that points the user in the general direction of the content.
Relevant content is outside of the field of view, and poor or no location guidance is provided to the user.
How to measure
Relevant content outside of the user field of view is discoverable through visual and/or audio cues.
When relevant content is outside the user's field of view, use directional indicators and audio cues to guide the user to the content. In many cases, a direct visual guide is preferred over directional arrows.
Directional indicators should not be built into the cursor.
A progress control provides feedback to the user that a long-running operation is underway. It can mean that the user cannot interact with the app when the progress indicator is visible and can also indicate how long the wait time might be.
Animated visual indicator, in the form of a progress bar or ring, showing progress during any data loading or processing. The visual indicator provides guidance on how long the wait could be.
User is informed that data loading is in progress, but there is no indication of how long the wait could be.
No data loading or process indicators for task taking longer than 5 seconds.
How to measure
During data loading verify there is no blank state for more than 5 seconds.
Provide a data loading animator showing progress in any situation when the user may perceive this app to have stalled or crashed. A reasonable rule of thumb is any 'loading' activity that could take more than 5 seconds.