Add LUIS results to Application Insights

This tutorial adds LUIS request and response information to Application Insights telemetry data storage. Once you have that data, you can query it with the Kusto language or PowerBi to analyze, aggregate, and report on intents, and entities of the utterance in real-time. This analysis helps you determine if you should add or edit the intents and entities of your LUIS app.

The bot is built with the Bot Framework 3.x and the Azure Web app bot.

In this tutorial, you learn how to:

  • Add Application Insights library to a web app bot
  • Capture and send LUIS query results to Application Insights
  • Query Application Insights for top intent, score, and utterance

Prerequisites

  • Your LUIS web app bot from the previous tutorial with Application Insights turned on.

Tip

If you do not already have a subscription, you can register for a free account.

All of the code in this tutorial is available on the LUIS-Samples github repository and each line associated with this tutorial is commented with //APPINSIGHT:.

Web app bot with LUIS

This tutorial assumes you have code that looks like the following or that you have completed the other tutorial:

/*-----------------------------------------------------------------------------
This template demonstrates how to use dialogs with a LuisRecognizer to add 
natural language support to a bot. 
For a complete walkthrough of creating this type of bot see the article at
https://aka.ms/abs-node-luis
-----------------------------------------------------------------------------*/

var restify = require('restify');
var builder = require('botbuilder');
var botbuilder_azure = require("botbuilder-azure");

// Setup Restify Server
var server = restify.createServer();
server.listen(process.env.port || process.env.PORT || 3978, function () {
   console.log('%s listening to %s', server.name, server.url); 
});
  
// Create chat connector for communicating with the Bot Framework Service
var connector = new builder.ChatConnector({
    appId: process.env.MicrosoftAppId,
    appPassword: process.env.MicrosoftAppPassword,
    openIdMetadata: process.env.BotOpenIdMetadata 
});

// Listen for messages from users 
server.post('/api/messages', connector.listen());

/*----------------------------------------------------------------------------------------
* Bot Storage: This is a great spot to register the private state storage for your bot. 
* We provide adapters for Azure Table, CosmosDb, SQL Azure, or you can implement your own!
* For samples and documentation, see: https://github.com/Microsoft/BotBuilder-Azure
* ---------------------------------------------------------------------------------------- */

var tableName = 'botdata';
var azureTableClient = new botbuilder_azure.AzureTableClient(tableName, process.env['AzureWebJobsStorage']);
var tableStorage = new botbuilder_azure.AzureBotStorage({ gzipData: false }, azureTableClient);

// Create your bot with a function to receive messages from the user
// This default message handler is invoked if the user's utterance doesn't
// match any intents handled by other dialogs.
var bot = new builder.UniversalBot(connector, function (session, args) {
    session.send('You reached the default message handler. You said \'%s\'.', session.message.text);
});

bot.set('storage', tableStorage);

// Make sure you add code to validate these fields
var luisAppId = process.env.LuisAppId;
var luisAPIKey = process.env.LuisAPIKey;
var luisAPIHostName = process.env.LuisAPIHostName || 'westus.api.cognitive.microsoft.com';

const LuisModelUrl = 'https://' + luisAPIHostName + '/luis/v2.0/apps/' + luisAppId + '?subscription-key=' + luisAPIKey;

// Create a recognizer that gets intents from LUIS, and add it to the bot
var recognizer = new builder.LuisRecognizer(LuisModelUrl);
bot.recognizer(recognizer);

// Add a dialog for each intent that the LUIS app recognizes.
// See https://docs.microsoft.com/en-us/bot-framework/nodejs/bot-builder-nodejs-recognize-intent-luis 
bot.dialog('TurnOnDialog',
    (session, args) => {
        // Resolve and store any HomeAutomation.Device entity passed from LUIS.
        var intent = args.intent;
        var device = builder.EntityRecognizer.findEntity(intent.entities, 'HomeAutomation.Device');

        // Turn on a specific device if a device entity is detected by LUIS
        if (device) {
            session.send('Ok, turning on the %s.', device.entity);
            // Put your code here for calling the IoT web service that turns on a device
        } else {
            // Assuming turning on lights is the default
            session.send('Ok, turning on the lights');
            // Put your code here for calling the IoT web service that turns on a device
        }
        session.endDialog();
    }
).triggerAction({
    matches: 'HomeAutomation.TurnOn'
})

bot.dialog('TurnOffDialog',
    (session, args) => {
        // Resolve and store any HomeAutomation.Device entity passed from LUIS.
        var intent = args.intent;
        var device = builder.EntityRecognizer.findEntity(intent.entities, 'HomeAutomation.Device');

        // Turn off a specific device if a device entity is detected by LUIS
        if (device) {
            session.send('Ok, turning off the %s.', device.entity);
            // Put your code here for calling the IoT web service that turns off a device
        } else {
            // Assuming turning off lights is the default
            session.send('Ok, turning off the lights.');
            // Put your code here for calling the IoT web service that turns off a device
        }
        session.endDialog();
    }
).triggerAction({
    matches: 'HomeAutomation.TurnOff'
})

Add Application Insights library to web app bot

Currently, the Application Insights service, used in this web app bot, collects general state telemetry for the bot. It does not collect LUIS request and response information that you need to check and fix your intents and entities.

In order to capture the LUIS request and response, the web app bot needs the Application Insights NPM package installed and configured in the app.js file. Then the intent dialog handlers need to send the LUIS request and response information to Application Insights.

  1. In the Azure portal, in the web app bot service, select Build under the Bot Management section.

    Search for app insights

  2. A new browser tab opens with the App Service Editor. Select the app name in the top bar, then select Open Kudu Console.

    Search for app insights

  3. In the console, enter the following command to install Application Insights and the Underscore packages:

    cd site\wwwroot && npm install applicationinsights && npm install underscore
    

    Search for app insights

    Wait for the packages to install:

    luisbot@1.0.0 D:\home\site\wwwroot
    `-- applicationinsights@1.0.1 
      +-- diagnostic-channel@0.2.0 
      +-- diagnostic-channel-publishers@0.2.1 
      `-- zone.js@0.7.6 
    
    npm WARN luisbot@1.0.0 No repository field.
    luisbot@1.0.0 D:\home\site\wwwroot
    +-- botbuilder-azure@3.0.4
    | `-- azure-storage@1.4.0
    |   `-- underscore@1.4.4 
    `-- underscore@1.8.3 
    

    You are done with the kudu console browser tab.

Capture and send LUIS query results to Application Insights

  1. In the App Service Editor browser tab, open the app.js file.

  2. Add the following NPM libraries under the existing requires lines:

    // APPINSIGHT: Add underscore for flattening to name/value pairs
    var _ = require("underscore");
    
    // APPINSIGHT: Add NPM package applicaitoninsights
    let appInsights = require("applicationinsights");
    
  3. Create the Application Insights object and use the web app bot application setting BotDevInsightsKey:

    // APPINSIGHT: Set up ApplicationInsights with Web App Bot settings "BotDevAppInsightsKey"
    appInsights.setup(process.env.BotDevAppInsightsKey)
        .setAutoDependencyCorrelation(true)
        .setAutoCollectRequests(true)
        .setAutoCollectPerformance(true)
        .setAutoCollectExceptions(true)
        .setAutoCollectDependencies(true)
        .setAutoCollectConsole(true,true)
        .setUseDiskRetryCaching(true)
        .start();
    
    // APPINSIGHT: Get client 
    let appInsightsClient = appInsights.defaultClient;
    
  4. Add the appInsightsLog function:

    // APPINSIGHT: Log LUIS results to Application Insights
    // APPINSIGHT: must flatten as name/value pairs
    var appInsightsLog = function(session,args) {
        
        // APPINSIGHT: put bot session and LUIS results into single object
        var data = Object.assign({}, session.message,args);
        
        // APPINSIGHT: ApplicationInsights Trace 
        console.log(data);
    
        // APPINSIGHT: Flatten data into name/value pairs
        flatten = function(x, result, prefix) {
            if(_.isObject(x)) {
                _.each(x, function(v, k) {
                    flatten(v, result, prefix ? prefix + '_' + k : k)
                })
            } else {
                result["LUIS_" + prefix] = x
            }
            return result;
        }
    
        // APPINSIGHT: call fn to flatten data
        var flattenedData = flatten(data, {})
    
        // APPINSIGHT: send data to Application Insights
        appInsightsClient.trackEvent({name: "LUIS-results", properties: flattenedData});
    }
    

    The last line of the function is where the data is added to Application Insights. The event's name is LUIS-results, a unique name apart from any other telemetry data collected by this web app bot.

  5. Use the appInsightsLog function. You add it to every intent dialog:

    // APPINSIGHT: Log results to Application Insights
    appInsightsLog(session,args);
    
  6. To test your web app bot, use the Test in Web Chat feature. You should see no difference because all the work is in Application Insights, not in the bot responses.

View LUIS entries in Application Insights

Open Application Insights to see the LUIS entries.

  1. In the portal, select All resources then filter by the web app bot name. Click on the resource with the type Application Insights. The icon for Application Insights is a light bulb.

    Search for app insights

  2. When the resource opens, click on the Search icon of the magnifying glass in the far right panel. A new panel to the right displays. Depending on how much telemetry data is found, the panel may take a second to display. Search for LUIS-results and hit enter on the keyboard. The list is narrowed to just LUIS query results added with this tutorial.

    Filter to dependencies

  3. Select the top entry. A new window displays more detailed data including the custom data for the LUIS query at the far-right. The data includes the top intent, and its score.

    Dependency details

    When you are done, select the far-right top X to return to the list of dependency items.

Tip

If you want to save the dependency list and return to it later, click on ...More and click Save favorite.

Query Application Insights for intent, score, and utterance

Application Insights gives you the power to query the data with the Kusto language, as well as export it to PowerBI.

  1. Click on Analytics at the top of the dependency listing, above the filter box.

    Analytics button

  2. A new window opens with a query window at the top and a data table window below that. If you have used databases before, this arrangement is familiar. The query includes all items from the last 24 hours beginning with the name LUIS-results. The CustomDimensions column has the LUIS query results as name/value pairs.

    Analytics query window

  3. To pull out the top intent, score, and utterance, add the following just above the last line in the query window:

    | extend topIntent = tostring(customDimensions.LUIS_intent_intent)
    | extend score = todouble(customDimensions.LUIS_intent_score)
    | extend utterance = tostring(customDimensions.LUIS_text)
    
  4. Run the query. Scroll to the far right in the data table. The new columns of topIntent, score, and utterance are available. Click on the topIntent column to sort.

    Analytics top intent

Learn more about the Kusto query language or export the data to PowerBi.

Next steps

Other information you may want to add to the application insights data includes app ID, version ID, last model change date, last train date, last publish date. These values can either be retrieved from the endpoint URL (app ID and version ID), or from an authoring API call then set in the web app bot settings and pulled from there.

If you are using the same endpoint subscription for more than one LUIS app, you should also include the subscription ID and a property stating that it is a shared key.