question

JohnHynes-5564 avatar image
0 Votes"
JohnHynes-5564 asked JohnHynes-5564 commented

Filter Logging for Azure Application Insights

I have created an App Service (.NET 5.0) that logs to Application Insights.

The logging is setup in my startup.cs as follows:

     public void ConfigureServices( IServiceCollection services )
     {
         services.AddLogging( builder =>
         {
             builder.AddApplicationInsights( Configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"] );
             builder.AddFilter<ApplicationInsightsLoggerProvider>( "Microsoft.AspNetCore.Routing.EndpointMiddleware", LogLevel.Information );
             builder.AddFilter<ApplicationInsightsLoggerProvider>( "Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker", LogLevel.Information );
             builder.AddFilter<ApplicationInsightsLoggerProvider>( "Microsoft.AspNetCore.Hosting.Diagnostics", LogLevel.Error );
             builder.AddFilter<ApplicationInsightsLoggerProvider>( "MyApp.ClientAPI.Controllers.AppointmentController", LogLevel.Debug );
             builder.AddFilter<ApplicationInsightsLoggerProvider>( "MyApp.ClientAPI", LogLevel.Debug );
             builder.AddFilter<ApplicationInsightsLoggerProvider>( "Microsoft", LogLevel.Error );
             builder.AddFilter<ApplicationInsightsLoggerProvider>( "", LogLevel.Warning );
             builder.AddFilter<ApplicationInsightsLoggerProvider>( null, LogLevel.Warning );
         } );
         services.AddApplicationInsightsTelemetry( Configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"] );

I also have the same settings in my appsettings.json:

 "Logging":
 {
     "LogLevel":
     {
         "Default": "Warning",
         "Microsoft": "Error",
         "Microsoft.Hosting.Lifetime": "Error",
         "Microsoft.AspNetCore.Hosting.Diagnostics": "Warning",
         "Logging:LogLevel:Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker": "Warning",
         "Logging:LogLevel:Microsoft.AspNetCore.Routing.EndpointMiddleware": "Warning",
         "MyApp.ClientAPI": "Debug",
         "MyApp.ClientAPI.Controllers": "Debug"
     }
 },

In the Azure Portal, on the "App Service Logs" page of the App Service, I have Application Logging (Blob) Level set to Verbose.

The log files contain verbose output from everywhere. The above settings are ignored.

If I try and query the logs in Application Insights to find my exceptions, for example using the query:

// Show logs with warnings or exceptions
// A list of logs which contain warnings or exceptions (latest logs shown first).
FunctionAppLogs
| where Level == "Warning" or Level == "Error"
| project TimeGenerated, HostInstanceId, Level, Message, _ResourceId
| sort by TimeGenerated desc

It just returns an error:
'where' operator: Failed to resolve table or column expression named 'FunctionAppLogs'

How can I create log files that contain debug messages from my code, but only warnings/errors from system code?

I just want to create a log file with my debug output without having to wade through thosands of lines of system debug messages?

azure-functionsazure-monitor
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

AnuragSingh-MSFT avatar image
0 Votes"
AnuragSingh-MSFT answered JohnHynes-5564 commented

@JohnHynes-5564

The filtering rule for logging is applied as mentioned here:

Therefore, the order of filters should be from generalized to more specialized category. Also, the logs level specified indicates the minimum level, i.e., if LogLevel is specified as "Warning" then logs with level Warning, Error and Critical are logged. More detailed are available here.

If you would like to generate logs that only contains Debug ( and higher) LogLevels from your code, but only warnings/errors (and Critical) from other code, you may modify your appsettings.json file as below:

 "Logging": {
     "LogLevel": {
         "Default": "Warning",
         "MyApp.ClientAPI": "Debug",
         "MyApp.ClientAPI.Controllers": "Debug"
     }
 }

Here,
• "Warning" specified for "Default" would apply for everything (Microsoft provided assemblies and other assemblies that might be getting referenced from Nuget package, project reference etc.) and
• "Debug" level for <MyApp.ClientAPI> and <MyApp.ClientAPI.Controllers> Log category.


Please refer to the article here for more Details on Log Category and ensure that the Log Category is correctly specified.
The above entries in appsettings.json would apply to all enabled logging providers except for the "Windows Event log".


To enable "ApplicationInsights" log provider, the following code can be included in "ConfigureServices" method:

         services.AddApplicationInsightsTelemetry(Configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"])

If you have already specified the log filters in appsettings.json, you don’t have to do anything else here (Please check the "Update: 08/06" section below for more details). Simply enabling ApplicationInsights log provider would apply the global settings from appsettings.json. However if you would like to override any filter or change settings specifically for "ApplicationInsights", you may use something like below:

    services.AddApplicationInsightsTelemetry(Configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"])
             services.AddLogging(builder =>
               {
                   builder.AddFilter<ApplicationInsightsLoggerProvider>("", LogLevel.Warning);
                   builder.AddFilter<ApplicationInsightsLoggerProvider>("MyApp.ClientAPI", "Debug");
                   builder.AddFilter<ApplicationInsightsLoggerProvider>("MyApp.ClientAPI.Controllers", "Debug")
               });

The code snippet applies the same settings as available in appsettings.json earlier, but here it applies to "ApplicationInsights" specifically. Ensure that the Log Category mentioned is accurate - Details


For querying data from ApplicationInsights, use the table "AppTraces" instead of "FunctionAppLogs". "FunctionAppLogs" will be used for logs generated by FunctionApps.


Update: 08/06
There are 2 parts to your queries - first related to Application Insights and other one related to App Service logs.
They both are not part of the shared framework and are enabled through NuGet. Therefore, the "default settings" under "Logging" without specific Provider do not apply to them as for other shared framework log providers. I tested out locally and with Azure App Service, and here is some updated information that should help you.
App Service Logs - Log filtering:
As you have also configured "App Service Logs" as mentioned in Logging in .NET Core and ASP.NET Core, please note that the setting filters in appsettings.json do not apply here. The options to control its settings is available in Azure portal from the "App Service | App Service Logs" blade as shown below.
121017-appservicelog.jpg
The "Level" mentioned here applies globally to all type of logs provider and log category. You may use the "Retention Period (Days)" option to control the retention period and hence limit the storage usage.

Application Insights - Log filtering:
Please note that the default log level for "ApplicationInsights" provider is set to "Warning" as mentioned here. In order for the log filtering to work with Application Insights, please edit the "Logging" in appsettings.json file as below:

 {
   "Logging": {
     "LogLevel": {
       "Default": "Warning",
       "Microsoft": "Error", 
       "MyApp": "Debug"
     },
     "ApplicationInsights": {
       "LogLevel": {
       "Default": "Warning",
       "Microsoft": "Error", 
       "MyApp": "Debug"
       }
     }
   },

Here, the "ApplicationInsights" alias applies the filtering to the logs collected by ApplicationInsights which is being saved in the associated workspace. More details are available here: Application Insights logging with .NET - Azure Monitor


Regarding the table names, prior to the introduction of workspace-based Application Insights resources, Application Insights data was stored separate from other log data in Azure Monitor. You will still see tables with Legacy Names under "Application Insights | Logs" blade. The new table names are available in "Log Analytics workspace | Logs" blade.
Therefore, you see "AppTraces" table under "AppIication Insights | Log" and "Traces" table under "Log Analytics workspace | Log". Please refer here for more details: Azure Monitor Application Insights workspace-based resource schema - Azure Monitor
You can also get the Tables available in respective sections by using the "Schema and Filter" option available in Logs section as shown below:
121111-schema.jpg




appservicelog.jpg (33.0 KiB)
schema.jpg (28.8 KiB)
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

That's very similar to what I had, apart from the code version having the AddFilter calls in reverse order.

I have tried switching the order but it made no difference.

I have removed all of the AddFilter settings in code, and reduced my appsettings.json to this:

     "Logging":
     {
         "LogLevel":
         {
             "Default": "Warning",
             "Microsoft": "Error",
             "MyApp": "Debug"
         }
     },

But still, if I look at the logs in the Blob container there are still thousands of lines of Microsoft Trace and Debug messages.

For querying, I have found I can use the traces table (there isn't an AppTraces table). This is helpful, but still doesn't resolve the issue that I'm storing huge amounts of logging information that I don't want to store.

0 Votes 0 ·

@JohnHynes-5564
Thank you for the update. I am sorry for the confusion in my previous post. After testing some settings and review, I have updated the answer above - please check it. I hope it helps.

0 Votes 0 ·

Thanks, the application insights are filtered now. I will turn of the app service logging as its not much use if I can't filter it.

0 Votes 0 ·