How to interact with a background app in Cortana (HTML)

Learn how a user can interact with a background app through the Cortana voice and canvas during the execution of a voice command.

Voice commands with Cortana can include a rich user experience and interaction flow within Cortana that is controlled by the background app. The app can specify a number of different types of screens to support functionality that includes:

• Successful completion
• Hand-off
• Progress
• Confirmation
• Disambiguation
• Error

Prerequisites

This topic builds on Launch a background app with voice commands in Cortana. We continue here to demonstrate features with a trip planning and management app named Adventure Works.

To complete this tutorial, have a look through these topics to get familiar with the technologies discussed here:

• Install Microsoft Visual Studio.
• Get a developer license. For instructions, see Develop using Visual Studio 2013.
• Create your first app using JavaScript.
• Roadmap for Windows Store apps using JavaScript
• Learn about events with Quickstart: adding HTML controls and handling events
• See the Voice Command Definition (VCD) elements and attributes v1.2 reference for more info about VCD elements.
• See Cortana design guidelines for how to integrate your app with Cortana and Speech design guidelines for helpful tips on designing a useful and engaging speech-enabled app.

Instructions

Step 1: Completion screen

A completion screen provides the user with information about the completed voice command task.

Here we show how Cortana can display a list of travel itinerary results from the Adventure Works app for upcoming trips to Las Vegas.

1. Choose the feedback strings to be displayed and spoken by Cortana

   Follow the Cortana design guidelines for recommendations on composing strings that Cortana shows and speaks.

2. Choose content tiles based on the action performed (optional)

   Content tiles can provide additional context for the user and help keep the feedback strings concise.

   Cortana supports the following content tile templates (only one template can be used on the completion screen):

   • Title only
   • Title with up to three lines of text
   • Title with icon
   • Title with icon and up to three lines of text

   The icon can be:

   • 68w x 68h
   • 68w x 92h
   • 280w x 140h

   You can also let users launch your app in the foreground by either tapping a tile or the text link to your app.

3. Show the successful completion screen

   Here's an example of a successful completion screen with multiple content tiles.

   N/A
   

Step 2: Hand-off screen

Once a voice command is recognized, Cortana must present feedback in approximately .5 seconds. If the app service cannot complete the action specified by the voice command within .5 seconds, Cortana presents the user with a hand-off screen for up to 5 seconds.

Here's an example of a hand-off screen for the Adventure Works app. In this example, a user has queried Cortana for upcoming flights to Las Vegas. The hand-off screen includes a message customized with the app service name, an icon, and the Feedback string declared in the VCD file.

Step 3: Progress screen

Once a voice command is recognized, Cortana must present feedback in approximately .5 seconds. If the app service requires more time to complete the action, it can provide a progress screen to inform the user that the voice command is being actively handled.

Cortana shows a progress screen for a maximum of 5 seconds. After 5 seconds, Cortana presents the user with an error message and the app service is closed. If the app service needs more than 5 seconds to complete the action, it can continue to update Cortana with progress screens.

Here's an example of a hand-off screen for the Adventure Works app. In this example, a user has canceled a trip to Las Vegas through Cortana. The progress screen includes a message customized for the action, an icon, and a content tile with information about the trip being canceled.

1. Choose the feedback strings to be displayed and spoken by Cortana

   Follow the Cortana design guidelines for recommendations on composing strings that Cortana shows and speaks.

2. Choose content tiles based on the action performed (optional)

   Content tiles can provide additional context for the user and help keep the feedback strings concise.

   Cortana supports the following content tile templates (only one template can be used on the completion screen):

   • Title only
   • Title with up to three lines of text
   • Title with icon
   • Title with icon and up to three lines of text

   The icon can be:

   • 68w x 68h
   • 68w x 92h
   • 280w x 140h

   You can also let users launch your app in the foreground by either tapping a tile or the text link to your app.

3. Build the response

   Call ReportProgressAsync to show the progress screen in Cortana.

4. Show the progress screen

   Here's an example of a progress screen with a content tile.

   N/A
   

Step 4: Confirmation screen

When an action specified by a voice command is irreversible, has a significant impact, or the recognition confidence is not high, an app service can request confirmation.

Here's an example of a confirmation screen for the Adventure Works app. In this example, a user has instructed the app service to cancel a trip to Las Vegas through Cortana. The app service has provided Cortana with a confirmation screen that prompts the user for a yes or no answer before canceling the trip.

If the user says something other than "Yes" or "No", Cortana cannot determine the answer to the question. In this case, Cortana prompts the user with a similar question provided by the app service.

On the second prompt, if the user still doesn’t say "Yes" or "No", Cortana prompts the user a third time with the same question prefixed with an apology. If the user still doesn’t say "Yes" or "No", Cortana stops listening for voice input and asks the user to tap one of the buttons instead.

The confirmation screen includes a message customized for the action, an icon, and a content tile with information about the trip being canceled.

1. Choose the feedback strings to be displayed and spoken by Cortana

   Follow the Cortana design guidelines for recommendations on composing strings that Cortana shows and speaks.

2. Choose content tiles based on the action performed (optional)

   Content tiles can provide additional context for the user and help keep the feedback strings concise.

   Cortana supports the following content tile templates (only one template can be used on the completion screen):

   • Title only
   • Title with up to three lines of text
   • Title with icon
   • Title with icon and up to three lines of text

   The icon can be:

   • 68w x 68h
   • 68w x 92h
   • 280w x 140h

   You can also let users launch your app in the foreground by either tapping a tile or the text link to your app.

3. Build the response

   Call RequestConfirmationAsync to show the confirmation screen in Cortana.

4. Show the confirmation screen

   Here's an example of a confirmation screen with a content tile.

   N/A
   

Step 5: Disambiguation screen

When an action specified by a voice command has more than one possible outcome, an app service can request more info from the user.

Here's an example of a disambiguation screen for the Adventure Works app. In this example, a user has instructed the app service to cancel a trip to Las Vegas through Cortana. However, the user has two trips to Las Vegas on different dates and the app service cannot complete the action without the user selecting the intended trip.

The app service provides Cortana with a disambiguation screen that prompts the user to make a selection from a list of matching trips, before it cancels any.

In this case, Cortana prompts the user with a similar question provided by the app service.

On the second prompt, if the user still doesn’t say something that can be used to identify the selection, Cortana prompts the user a third time with the same question prefixed with an apology. If the user still doesn’t say something that can be used to identify the selection, Cortana stops listening for voice input and asks the user to tap one of the buttons instead.

The disambiguation screen includes a message customized for the action, an icon, and a content tile with information about the trip being canceled.

1. Choose the feedback strings to be displayed and spoken by Cortana

   Follow the Cortana design guidelines for recommendations on composing strings that Cortana shows and speaks.

2. Choose content tiles based on the action performed (optional)

   Content tiles can provide additional context for the user and help keep the feedback strings concise.

   Cortana supports the following content tile templates (only one template can be used on the completion screen):

   • Title only
   • Title with up to three lines of text
   • Title with icon
   • Title with icon and up to three lines of text

   The icon can be:

   • 68w x 68h
   • 68w x 92h
   • 280w x 140h

   You can also let users launch your app in the foreground by either tapping a tile or the text link to your app.

3. Build the response

   Call RequestDisambiguationAsync to show the disambiguation screen in Cortana.

4. Show the disambiguation screen

   Here's an example of a disambiguation screen with content tiles.

   N/A
   

Step 6: Error screen

When an action specified by a voice command cannot be completed, an app service can provide an error screen.

Here's an example of an error screen for the Adventure Works app. In this example, a user has instructed the app service to cancel a trip to Las Vegas through Cortana. However, the user does not have any trips scheduled to Las Vegas.

The app service provides Cortana with an error screen that includes a message customized for the action, an icon, and the specific error message.

1. Choose the feedback strings to be displayed and spoken by Cortana

   Follow the Cortana design guidelines for recommendations on composing strings that Cortana shows and speaks.

2. Choose content tiles based on the action performed (optional)

   Content tiles can provide additional context for the user and help keep the feedback strings concise.

   Cortana supports the following content tile templates (only one template can be used on the completion screen):

   • Title only
   • Title with up to three lines of text
   • Title with icon
   • Title with icon and up to three lines of text

   The icon can be:

   • 68w x 68h
   • 68w x 92h
   • 280w x 140h

   You can also let users launch your app in the foreground by either tapping a tile or the text link to your app.

3. Build the response

   Call ReportFailureAsync to show the error screen in Cortana.

4. Show the error screen

   Here's an example of an error screen.

   N/A
   

Complete example

Related topics

Cortana interactions

Launch a background app with voice commands in Cortana

VCD elements and attributes v1.2

Designers

Cortana design guidelines