Filtering and preprocessing telemetry in the Application Insights SDK

You can write and configure plug-ins for the Application Insights SDK to customize how telemetry can be enriched and processed before it's sent to the Application Insights service.

  • Sampling reduces the volume of telemetry without affecting your statistics. It keeps together related data points so that you can navigate between them when diagnosing a problem. In the portal, the total counts are multiplied to compensate for the sampling.
  • Filtering with Telemetry Processors lets you filter out telemetry in the SDK before it is sent to the server. For example, you could reduce the volume of telemetry by excluding requests from robots. Filtering is a more basic approach to reducing traffic than sampling. It allows you more control over what is transmitted, but you have to be aware that it affects your statistics - for example, if you filter out all successful requests.
  • Telemetry Initializers add or modify properties to any telemetry sent from your app, including telemetry from the standard modules. For example, you could add calculated values; or version numbers by which to filter the data in the portal.
  • The SDK API is used to send custom events and metrics.

Before you start:

Filtering

This technique gives you direct control over what is included or excluded from the telemetry stream. Filtering can be used to drop telemetry items from being sent to Application Insights. You can use it in conjunction with Sampling, or separately.

To filter telemetry, you write a telemetry processor and register it with the TelemetryConfiguration. All telemetry goes through your processor, and you can choose to drop it from the stream or give it to the next processor in the chain. This includes telemetry from the standard modules such as the HTTP request collector and the dependency collector, and telemetry you have tracked yourself. You can, for example, filter out telemetry about requests from robots, or successful dependency calls.

Warning

Filtering the telemetry sent from the SDK using processors can skew the statistics that you see in the portal, and make it difficult to follow related items.

Instead, consider using sampling.

Create a telemetry processor (C#)

  1. To create a filter, implement ITelemetryProcessor.

    Notice that Telemetry Processors construct a chain of processing. When you instantiate a telemetry processor, you are given a reference to the next processor in the chain. When a telemetry data point is passed to the Process method, it does its work and then calls (or not calls) the next Telemetry Processor in the chain.

    using Microsoft.ApplicationInsights.Channel;
    using Microsoft.ApplicationInsights.Extensibility;
    
    public class SuccessfulDependencyFilter : ITelemetryProcessor
    {
        private ITelemetryProcessor Next { get; set; }
    
        // next will point to the next TelemetryProcessor in the chain.
        public SuccessfulDependencyFilter(ITelemetryProcessor next)
        {
            this.Next = next;
        }
    
        public void Process(ITelemetry item)
        {
            // To filter out an item, return without calling the next processor.
            if (!OKtoSend(item)) { return; }
    
            this.Next.Process(item);
        }
    
        // Example: replace with your own criteria.
        private bool OKtoSend (ITelemetry item)
        {
            var dependency = item as DependencyTelemetry;
            if (dependency == null) return true;
    
            return dependency.Success != true;
        }
    }
    
  2. Add your processor.

ASP.NET apps Insert this snippet in ApplicationInsights.config:

<TelemetryProcessors>
  <Add Type="WebApplication9.SuccessfulDependencyFilter, WebApplication9">
	 <!-- Set public property -->
	 <MyParamFromConfigFile>2-beta</MyParamFromConfigFile>
  </Add>
</TelemetryProcessors>

You can pass string values from the .config file by providing public named properties in your class.

Warning

Take care to match the type name and any property names in the .config file to the class and property names in the code. If the .config file references a non-existent type or property, the SDK may silently fail to send any telemetry.

Alternatively, you can initialize the filter in code. In a suitable initialization class - for example AppStart in Global.asax.cs - insert your processor into the chain:

var builder = TelemetryConfiguration.Active.DefaultTelemetrySink.TelemetryProcessorChainBuilder;
builder.Use((next) => new SuccessfulDependencyFilter(next));

// If you have more processors:
builder.Use((next) => new AnotherProcessor(next));

builder.Build();

TelemetryClients created after this point will use your processors.

ASP.NET Core/ Worker Service apps

Note

Adding processor using ApplicationInsights.config or using TelemetryConfiguration.Active is not valid for ASP.NET Core applications or if you are using Microsoft.ApplicationInsights.WorkerService SDK.

For apps written using ASP.NET Core or WorkerService, adding a new TelemetryProcessor is done by using AddApplicationInsightsTelemetryProcessor extension method on IServiceCollection, as shown below. This method is called in ConfigureServices method of your Startup.cs class.

    public void ConfigureServices(IServiceCollection services)
    {
        // ...
        services.AddApplicationInsightsTelemetry();
        services.AddApplicationInsightsTelemetryProcessor<SuccessfulDependencyFilter>();

        // If you have more processors:
        services.AddApplicationInsightsTelemetryProcessor<AnotherProcessor>();
    }

Example filters

Synthetic requests

Filter out bots and web tests. Although Metrics Explorer gives you the option to filter out synthetic sources, this option reduces traffic and ingestion size by filtering them at the SDK itself.

public void Process(ITelemetry item)
{
  if (!string.IsNullOrEmpty(item.Context.Operation.SyntheticSource)) {return;}

  // Send everything else:
  this.Next.Process(item);
}

Failed authentication

Filter out requests with a "401" response.

public void Process(ITelemetry item)
{
    var request = item as RequestTelemetry;

    if (request != null &&
    request.ResponseCode.Equals("401", StringComparison.OrdinalIgnoreCase))
    {
        // To filter out an item, return without calling the next processor.
        return;
    }

    // Send everything else
    this.Next.Process(item);
}

Filter out fast remote dependency calls

If you only want to diagnose calls that are slow, filter out the fast ones.

Note

This will skew the statistics you see on the portal.

public void Process(ITelemetry item)
{
    var request = item as DependencyTelemetry;

    if (request != null && request.Duration.TotalMilliseconds < 100)
    {
        return;
    }
    this.Next.Process(item);
}

Diagnose dependency issues

This blog describes a project to diagnose dependency issues by automatically sending regular pings to dependencies.

JavaScript Web applications

Filtering using ITelemetryInitializer

  1. Create a telemetry initializer callback function. The callback function takes ITelemetryItem as a parameter, which is the event that is being processed. Returning false from this callback results in the telemetry item to be filtered out.

    var filteringFunction = (envelope) => {
      if (envelope.data.someField === 'tobefilteredout') {
      	return false;
      }
    
      return true;
    };
    
  2. Add your telemetry initializer callback:

    appInsights.addTelemetryInitializer(filteringFunction);
    

Add/modify properties: ITelemetryInitializer

Use telemetry initializers to enrich telemetry with additional information and/or to override telemetry properties set by the standard telemetry modules.

For example, the Application Insights for Web package collect telemetry about HTTP requests. By default, it flags as failed any request with a response code >= 400. But if you want to treat 400 as a success, you can provide a telemetry initializer that sets the Success property.

If you provide a telemetry initializer, it is called whenever any of the Track*() methods are called. This includes Track() methods called by the standard telemetry modules. By convention, these modules do not set any property that has already been set by an initializer. Telemetry initializers are called before calling telemetry processors. So any enrichments done by initializers are visible to processors.

Define your initializer

C#

using System;
using Microsoft.ApplicationInsights.Channel;
using Microsoft.ApplicationInsights.DataContracts;
using Microsoft.ApplicationInsights.Extensibility;

namespace MvcWebRole.Telemetry
{
  /*
   * Custom TelemetryInitializer that overrides the default SDK
   * behavior of treating response codes >= 400 as failed requests
   *
   */
  public class MyTelemetryInitializer : ITelemetryInitializer
  {
	public void Initialize(ITelemetry telemetry)
	{
		var requestTelemetry = telemetry as RequestTelemetry;
		// Is this a TrackRequest() ?
		if (requestTelemetry == null) return;
		int code;
		bool parsed = Int32.TryParse(requestTelemetry.ResponseCode, out code);
		if (!parsed) return;
		if (code >= 400 && code < 500)
		{
			// If we set the Success property, the SDK won't change it:
			requestTelemetry.Success = true;

			// Allow us to filter these requests in the portal:
			requestTelemetry.Properties["Overridden400s"] = "true";
		}
		// else leave the SDK to set the Success property
	}
  }
}

ASP.NET apps: Load your initializer

In ApplicationInsights.config:

<ApplicationInsights>
  <TelemetryInitializers>
	<!-- Fully qualified type name, assembly name: -->
	<Add Type="MvcWebRole.Telemetry.MyTelemetryInitializer, MvcWebRole"/>
	...
  </TelemetryInitializers>
</ApplicationInsights>

Alternatively, you can instantiate the initializer in code, for example in Global.aspx.cs:

protected void Application_Start()
{
    // ...
    TelemetryConfiguration.Active.TelemetryInitializers.Add(new MyTelemetryInitializer());
}

See more of this sample.

ASP.NET Core/ Worker Service apps: Load your initializer

Note

Adding initializer using ApplicationInsights.config or using TelemetryConfiguration.Active is not valid for ASP.NET Core applications or if you are using Microsoft.ApplicationInsights.WorkerService SDK.

For apps written using ASP.NET Core or WorkerService, adding a new TelemetryInitializer is done by adding it to the Dependency Injection container, as shown below. This is done in Startup.ConfigureServices method.

 using Microsoft.ApplicationInsights.Extensibility;
 using CustomInitializer.Telemetry;
 public void ConfigureServices(IServiceCollection services)
{
    services.AddSingleton<ITelemetryInitializer, MyTelemetryInitializer>();
}

Java telemetry initializers

Java SDK documentation

public interface TelemetryInitializer
{ /** Initializes properties of the specified object. * @param telemetry The {@link com.microsoft.applicationinsights.telemetry.Telemetry} to initialize. */

void initialize(Telemetry telemetry); }

Then register the custom initializer in your applicationinsights.xml file.

<Add type="mypackage.MyConfigurableContextInitializer">
    <Param name="some_config_property" value="some_value" />
</Add>

JavaScript telemetry initializers

JavaScript

Insert a telemetry initializer immediately after the initialization code that you got from the portal:

<script type="text/javascript">
	// ... initialization code
	...({
		instrumentationKey: "your instrumentation key"
	});
	window.appInsights = appInsights;


	// Adding telemetry initializer.
	// This is called whenever a new telemetry item
	// is created.

	appInsights.queue.push(function () {
		appInsights.context.addTelemetryInitializer(function (envelope) {
			var telemetryItem = envelope.data.baseData;

			// To check the telemetry items type - for example PageView:
			if (envelope.name == Microsoft.ApplicationInsights.Telemetry.PageView.envelopeType) {
				// this statement removes url from all page view documents
				telemetryItem.url = "URL CENSORED";
			}

			// To set custom properties:
			telemetryItem.properties = telemetryItem.properties || {};
			telemetryItem.properties["globalProperty"] = "boo";

			// To set custom metrics:
			telemetryItem.measurements = telemetryItem.measurements || {};
			telemetryItem.measurements["globalMetric"] = 100;
		});
	});

	// End of inserted code.

	appInsights.trackPageView();
</script>

For a summary of the non-custom properties available on the telemetryItem, see Application Insights Export Data Model.

You can add as many initializers as you like, and they are called in the order they are added.

Example TelemetryInitializers

Add custom property

The following sample initializer adds a custom property to every tracked telemetry.

public void Initialize(ITelemetry item)
{
  var itemProperties = item as ISupportProperties;
  if(itemProperties != null && !itemProperties.ContainsKey("customProp"))
    {
        itemProperties.Properties["customProp"] = "customValue";
    }
}

Add cloud role name

The following sample initializer sets cloud role name to every tracked telemetry.

public void Initialize(ITelemetry telemetry)
{
    if(string.IsNullOrEmpty(telemetry.Context.Cloud.RoleName))
    {
        telemetry.Context.Cloud.RoleName = "MyCloudRoleName";
    }
}

ITelemetryProcessor and ITelemetryInitializer

What's the difference between telemetry processors and telemetry initializers?

  • There are some overlaps in what you can do with them: both can be used to add or modify properties of telemetry, though it is recommended to use initializers for that purpose.
  • TelemetryInitializers always run before TelemetryProcessors.
  • TelemetryInitializers may be called more than once. By convention, they do not set any property that has already been set.
  • TelemetryProcessors allow you to completely replace or discard a telemetry item.
  • All registered TelemetryInitializers are guaranteed to be called for every telemetry item. For Telemetry processors, SDK guarantees calling the very first telemetry processor. Whether the rest of the processors are called or not, is decided by the preceding telemetry processors.
  • Use TelemetryInitializers to enrich telemetry with additional properties, or override existing one. Use TelemetryProcessor to filter out telemetry.

Troubleshooting ApplicationInsights.config

  • Confirm that the fully qualified type name and assembly name are correct.
  • Confirm that the applicationinsights.config file is in your output directory and contains any recent changes.

Reference docs

SDK Code

Next steps