Persist all releated to exceptions metrics, while sampling everything else

Siarhei Kharlap 0 Reputation points
2024-05-15T08:55:09.4533333+00:00

We are trying to collect all metrics related to exceptions (request, dependency, trace) without sampling) without sampling, to provide a better debug experience.

According to https://learn.microsoft.com/en-us/azure/azure-monitor/app/sampling-classic-api
User's image

It seems like it is enough to customise Exception event sampling rate.
However))

What we have tried:

Try 1

  1. Fixed-rate sampling for exceptions set to 99.9999 (internal SDK stuff which ignore 100)
  2. Disabled sampling for other metrics (relying on ingesting)
  3. Ingesting (AI level sampling) set to 2%.
    Results
    Every Exception Request Trace and Dependency are visible in 'Live Metrics'
    Every Exception is persisted and is visible in the Transaction search
    Only 2% of related metrics are visible in the Transaction search which affects End-To-End transactions (you are able to see the expected full request history only in this 2% cases)
    And official monitor request against the log shows that.

union requests,dependencies,pageViews,browserTimings,exceptions,traces

| where timestamp > ago(1d)

| summarize RetainedPercentage = 100/avg(itemCount) by bin(timestamp, 1h), itemType
User's image

Try 2

  1. Default adaptive sampling
  2. Custome ITelemetryInitializer
public class ExceptionTelemetrySamplingFilter : ITelemetryInitializer
{
    public void Initialize(ITelemetry telemetry)
    {
        if (telemetry is ExceptionTelemetry)
        {
            ((ISupportSampling)telemetry).SamplingPercentage = 99.99999;
        }
    }
}
  1. Ingesting (AI level sampling) set to 2%.
    The results are the same as for try 1.

Try 3

  1. Fixed-rate sampling for exceptions set to 99.9999 (internal SDK stuff which ignores 100)
  2. Fixed-rate sampling for others set to 2 (prevent ingesting)
    The result is the same as for previous, except that now in 'Live Metrics' we see only Exceptions and others with 2% chance.

Any suggestions so it can be achieved?

Azure Monitor
Azure Monitor
An Azure service that is used to collect, analyze, and act on telemetry data from Azure and on-premises environments.
2,871 questions
ASP.NET Core
ASP.NET Core
A set of technologies in the .NET Framework for building web applications and XML web services.
4,246 questions
0 comments No comments
{count} votes