Tutorial: Fix unsure predictions by reviewing endpoint utterances

In this tutorial, improve app predictions by verifying or correcting utterances received via the LUIS HTTPS endpoint that LUIS is unsure of. Some utterances may have to be verified for intent and others may need to be verified for entity. You should review endpoint utterances as a regular part of you scheduled LUIS maintenance.

This review process is another way for LUIS to learn your app domain. LUIS selected the utterances that appear in the review list. This list is:

  • Specific to the app.
  • Is meant to improve the app's prediction accuracy.
  • Should be reviewed on a periodic basis.

By reviewing the endpoint utterances, you verify or correct the utterance's predicted intent. You also label custom entities that were not predicted or predicted incorrectly.

In this tutorial, you learn how to:

  • Import example app
  • Review endpoint utterances
  • Update phrase list
  • Train app
  • Publish app
  • Query endpoint of app to see LUIS JSON response

For this article, you can use the free LUIS account in order to author your LUIS application.

Import example app

Continue with the app created in the last tutorial, named HumanResources.

Use the following steps:

  1. Download and save app JSON file.

  2. Import the JSON into a new app.

  3. From the Manage section, on the Versions tab, clone the version, and name it review. Cloning is a great way to play with various LUIS features without affecting the original version. Because the version name is used as part of the URL route, the name can't contain any characters that are not valid in a URL.

  4. Train and publish the new app.

  5. Use the endpoint to add the following utterances. You can either do this with a script or from the endpoint in a browser. The utterances to add are:

    /* all utterances up through the Sentiment tutorial in series */
    const endpointUtterances = [
        "I'm looking for a job with Natural Language Processing",
        "I want to cancel on March 3",
        "When were HRF-123456 and hrf-234567 published in the last year?",
        "shift 123-45-6789 from Z-1242 to T-54672",
        "Please relocation jill-jones@mycompany.com from x-2345 to g-23456",
        "Here is my c.v. for the programmer job",
        "This is the lead welder paperwork.",
        "does form hrf-123456 cover the new dental benefits and medical plan",
        "Jill Jones work with the media team on the public portal was amazing",
    ];
    

    If you have all the versions of the app, through the series of tutorials, you may be surprised to see that the Review endpoint utterances list doesn't change, based on the version. There is a single pool of utterances to review, regardless of which version you are actively editing or which version of the app was published at the endpoint.

Review endpoint utterances

  1. Make sure your Human Resources app is in the Build section of LUIS. You can change to this section by selecting Build on the top, right menu bar.

  2. Select Review endpoint utterances from the left navigation. The list is filtered for the ApplyForJob intent.

    Screenshot of Review endpoint utterances button in left navigation

  3. Toggle the Entities view to see the labeled entities.

    Screenshot of Review endpoint utterances with Entities view toggle highlighted

    This utterance, I'm looking for a job with Natural Language Processing, is not in the correct intent.

    The reason the utterance was mispredicted is that the ApplyForJob intent has 21 utterances compared to the 7 utterances in GetJobInformation. The intent with more utterances will have a higher prediction. It is important that the quantity and quality of the utterances across intents is balanced.

  4. To align this utterance, select the correct intent and mark the Job entity within it. Add the changed utterance to the app by selecting the green checkbox.

    Utterance Correct intent Missing entities
    I'm looking for a job with Natural Language Processing GetJobInfo Job - "Natural Language Process"

    Adding the utterance moves the utterance from the Review endpoint utterances to the GetJobInformation intent. The endpoint utterance is now an example utterance for that intent.

    Along with aligning this utterance correctly, more utterances should be added to the GetJobInformation intent. That is left as an exercise for you to complete on your own. Each intent, except for the None intent, should have roughly the same number of example utterances. The None intent should have 10% of the total utterances in the app.

  5. Review the remaining utterances in this intent, labeling utterances and correcting the Aligned intent, if these are incorrect.

  6. The list should no longer have those utterances. If more utterances appear, continue to work through the list, correcting intents and labeling any missing entities, until the list is empty.

  7. Select the next intent in the Filter list, then continue correcting utterances and labeling entities. Remember the last step of each intent is to either select Add to aligned intent on the utterance row or check the box by each intent and select Add selected above the table.

    Continue until all intents and entities in the filter list have an empty list. This is a very small app. The review process takes only a few minutes.

Update phrase list

Keep the phrase list current with any newly discovered job names.

  1. Select Phrase lists from left navigation.

  2. Select the Jobs phrase list.

  3. Add Natural Language Processing as a value then select Save.

Train

LUIS doesn't know about the changes until it is trained.

  1. In the top right side of the LUIS website, select the Train button.

    Train button

  2. Training is complete when you see the green status bar at the top of the website confirming success.

    Trained status bar

Publish

If you imported this app, you need to select Sentiment analysis.

In order to receive a LUIS prediction in a chat bot or other client application, you need to publish the app to the endpoint.

  1. Select Publish in the top right navigation.

    LUIS publish to endpoint button in top right menu

  2. Select the Production slot and the Publish button.

    LUIS publish to endpoint

  3. Publishing is complete when you see the green status bar at the top of the website confirming success.

    LUIS publish to endpoint

  4. Select the endpoints link in the green status bar to go to the Keys and endpoints page. The endpoint URLs are listed at the bottom.

Get intent and entities from endpoint

Try an utterance close to the corrected utterance.

  1. In the Manage section (top right menu), on the Keys and endpoints page (left menu), select the endpoint URL at the bottom of the page. This action opens another browser tab with the endpoint URL in the address bar.

    The endpoint URL looks like https://<region>.api.cognitive.microsoft.com/luis/v2.0/apps/<appID>?verbose=true&subscription-key=<YOUR_KEY>&<optional-name-value-pairs>&q=<user-utterance-text>.

  2. Go to the end of the URL in the address and enter Are there any natural language processing jobs in my department right now?. The last querystring parameter is q, the utterance query.

    {
     "query": "are there any natural language processing jobs in my department right now?",
     "topScoringIntent": {
       "intent": "GetJobInformation",
       "score": 0.9247605
     },
     "intents": [
       {
         "intent": "GetJobInformation",
         "score": 0.9247605
       },
       {
         "intent": "ApplyForJob",
         "score": 0.129989788
       },
       {
         "intent": "FindForm",
         "score": 0.006438211
       },
       {
         "intent": "EmployeeFeedback",
         "score": 0.00408575451
       },
       {
         "intent": "Utilities.StartOver",
         "score": 0.00194211153
       },
       {
         "intent": "None",
         "score": 0.00166400627
       },
       {
         "intent": "Utilities.Help",
         "score": 0.00118593348
       },
       {
         "intent": "MoveEmployee",
         "score": 0.0007885918
       },
       {
         "intent": "Utilities.Cancel",
         "score": 0.0006373631
       },
       {
         "intent": "Utilities.Stop",
         "score": 0.0005980781
       },
       {
         "intent": "Utilities.Confirm",
         "score": 3.719905E-05
       }
     ],
     "entities": [
       {
         "entity": "right now",
         "type": "builtin.datetimeV2.datetime",
         "startIndex": 64,
         "endIndex": 72,
         "resolution": {
           "values": [
             {
               "timex": "PRESENT_REF",
               "type": "datetime",
               "value": "2018-07-05 15:23:18"
             }
           ]
         }
       },
       {
         "entity": "natural language processing",
         "type": "Job",
         "startIndex": 14,
         "endIndex": 40,
         "score": 0.9869922
       },
       {
         "entity": "natural language processing jobs",
         "type": "builtin.keyPhrase",
         "startIndex": 14,
         "endIndex": 45
       },
       {
         "entity": "department",
         "type": "builtin.keyPhrase",
         "startIndex": 53,
         "endIndex": 62
       }
     ],
     "sentimentAnalysis": {
       "label": "positive",
       "score": 0.8251864
     }
    }
    }
    

    The correct intent was predicted with a high score and the Job entity is detected as natural language processing.

Can reviewing be replaced by adding more utterances?

You may wonder why not add more example utterances. What is the purpose of reviewing endpoint utterances? In a real-world LUIS app, the endpoint utterances are from users with word choice and arrangement you haven't used yet. If you had used the same word choice and arrangement, the original prediction would have a higher percentage.

Why is the top intent on the utterance list?

Some of the endpoint utterances will have a high prediction score in the review list. You still need to review and verify those utterances. They are on the list because the next highest intent had a score too close to the top intent score. You want about 15% difference between the two top intents.

Clean up resources

When no longer needed, delete the LUIS app. To do so, select My apps from the top left menu. Select the ellipsis (...) to the right of the app name in the app list, select Delete. On the pop-up dialog Delete app?, select Ok.

Next steps

In this tutorial, you reviewed utterances submitted at the endpoint, that LUIS was unsure of. Once these utterances have been verified and moved into the correct intents as example utterances, LUIS will improve the prediction accuracy.