Filter Logging for Azure Application Insights

Question

I have created an App Service (.NET 5.0) that logs to Application Insights.

The logging is setup in my startup.cs as follows:

    public void ConfigureServices( IServiceCollection services )
    {
        services.AddLogging( builder =>
        {
            builder.AddApplicationInsights( Configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"] );
            builder.AddFilter( "Microsoft.AspNetCore.Routing.EndpointMiddleware", LogLevel.Information );
            builder.AddFilter( "Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker", LogLevel.Information );
            builder.AddFilter( "Microsoft.AspNetCore.Hosting.Diagnostics", LogLevel.Error );
            builder.AddFilter( "MyApp.ClientAPI.Controllers.AppointmentController", LogLevel.Debug );
            builder.AddFilter( "MyApp.ClientAPI", LogLevel.Debug );
            builder.AddFilter( "Microsoft", LogLevel.Error );
            builder.AddFilter( "", LogLevel.Warning );
            builder.AddFilter( null, LogLevel.Warning );
        } );
        services.AddApplicationInsightsTelemetry( Configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"] );

I also have the same settings in my appsettings.json:

"Logging":
{
    "LogLevel":
    {
        "Default": "Warning",
        "Microsoft": "Error",
        "Microsoft.Hosting.Lifetime": "Error",
        "Microsoft.AspNetCore.Hosting.Diagnostics": "Warning",
        "Logging:LogLevel:Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker": "Warning",
        "Logging:LogLevel:Microsoft.AspNetCore.Routing.EndpointMiddleware": "Warning",
        "MyApp.ClientAPI": "Debug",
        "MyApp.ClientAPI.Controllers": "Debug"
    }
},

In the Azure Portal, on the "App Service Logs" page of the App Service, I have Application Logging (Blob) Level set to Verbose.

The log files contain verbose output from everywhere. The above settings are ignored.

If I try and query the logs in Application Insights to find my exceptions, for example using the query:

// Show logs with warnings or exceptions
// A list of logs which contain warnings or exceptions (latest logs shown first).
FunctionAppLogs
| where Level == "Warning" or Level == "Error"
| project TimeGenerated, HostInstanceId, Level, Message, _ResourceId
| sort by TimeGenerated desc

It just returns an error:
'where' operator: Failed to resolve table or column expression named 'FunctionAppLogs'

How can I create log files that contain debug messages from my code, but only warnings/errors from system code?

I just want to create a log file with my debug output without having to wade through thosands of lines of system debug messages?

Accepted Answer

@John Hynes

The filtering rule for logging is applied as mentioned here:

Therefore, the order of filters should be from generalized to more specialized category. Also, the logs level specified indicates the minimum level, i.e., if LogLevel is specified as "Warning" then logs with level Warning, Error and Critical are logged. More detailed are available here.

If you would like to generate logs that only contains Debug ( and higher) LogLevels from your code, but only warnings/errors (and Critical) from other code, you may modify your appsettings.json file as below:

"Logging": {  
    "LogLevel": {  
        "Default": "Warning",  
        "MyApp.ClientAPI": "Debug",  
        "MyApp.ClientAPI.Controllers": "Debug"  
    }  
}

Here,
• "Warning" specified for "Default" would apply for everything (Microsoft provided assemblies and other assemblies that might be getting referenced from Nuget package, project reference etc.) and
• "Debug" level for and Log category.

Please refer to the article here for more Details on Log Category and ensure that the Log Category is correctly specified.
The above entries in appsettings.json would apply to all enabled logging providers except for the "Windows Event log".

To enable "ApplicationInsights" log provider, the following code can be included in "ConfigureServices" method:

        services.AddApplicationInsightsTelemetry(Configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"])

If you have already specified the log filters in appsettings.json, you don’t have to do anything else here (Please check the "Update: 08/06" section below for more details). Simply enabling ApplicationInsights log provider would apply the global settings from appsettings.json. However if you would like to override any filter or change settings specifically for "ApplicationInsights", you may use something like below:

   services.AddApplicationInsightsTelemetry(Configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"])  
            services.AddLogging(builder =>  
              {  
                  builder.AddFilter("", LogLevel.Warning);  
                  builder.AddFilter("MyApp.ClientAPI", "Debug");  
                  builder.AddFilter("MyApp.ClientAPI.Controllers", "Debug")  
              });

The code snippet applies the same settings as available in appsettings.json earlier, but here it applies to "ApplicationInsights" specifically. Ensure that the Log Category mentioned is accurate - Details

For querying data from ApplicationInsights, use the table "AppTraces" instead of "FunctionAppLogs". "FunctionAppLogs" will be used for logs generated by FunctionApps.

Update: 08/06
There are 2 parts to your queries - first related to Application Insights and other one related to App Service logs.
They both are not part of the shared framework and are enabled through NuGet. Therefore, the "default settings" under "Logging" without specific Provider do not apply to them as for other shared framework log providers. I tested out locally and with Azure App Service, and here is some updated information that should help you.
App Service Logs - Log filtering:
As you have also configured "App Service Logs" as mentioned in Logging in .NET Core and ASP.NET Core, please note that the setting filters in appsettings.json do not apply here. The options to control its settings is available in Azure portal from the "App Service | App Service Logs" blade as shown below.

The "Level" mentioned here applies globally to all type of logs provider and log category. You may use the "Retention Period (Days)" option to control the retention period and hence limit the storage usage.

Application Insights - Log filtering:
Please note that the default log level for "ApplicationInsights" provider is set to "Warning" as mentioned here. In order for the log filtering to work with Application Insights, please edit the "Logging" in appsettings.json file as below:

{  
  "Logging": {  
    "LogLevel": {  
      "Default": "Warning",  
      "Microsoft": "Error",   
      "MyApp": "Debug"  
    },  
    "ApplicationInsights": {  
      "LogLevel": {  
      "Default": "Warning",  
      "Microsoft": "Error",   
      "MyApp": "Debug"  
      }  
    }  
  },

Here, the "ApplicationInsights" alias applies the filtering to the logs collected by ApplicationInsights which is being saved in the associated workspace. More details are available here: Application Insights logging with .NET - Azure Monitor

Regarding the table names, prior to the introduction of workspace-based Application Insights resources, Application Insights data was stored separate from other log data in Azure Monitor. You will still see tables with Legacy Names under "Application Insights | Logs" blade. The new table names are available in "Log Analytics workspace | Logs" blade.
Therefore, you see "AppTraces" table under "AppIication Insights | Log" and "Traces" table under "Log Analytics workspace | Log". Please refer here for more details: Azure Monitor Application Insights workspace-based resource schema - Azure Monitor
You can also get the Tables available in respective sections by using the "Schema and Filter" option available in Logs section as shown below:

Filter Logging for Azure Application Insights

0 additional answers