question

CosminStirbu-1831 avatar image
0 Votes"
CosminStirbu-1831 asked YanivLavi-3333 commented

Stateful custom log alerts not resolved

Hello,

We are trying out the preview stateful custom log alerts feature.

For some reason, our alerts are fired as expected but they are not resolved.

I have attached a screenshot with how the alert is configured.

110509-screenshot-2021-06-30-at-124616.png

Thank you,
Cosmin


azure-monitor
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thanks for the post, @CosminStirbu-1831. Since this is a preview service, checking with the Azure monitor feature team on how to reconcile state-ful custom log alerts.
Please stay tuned. Will share an update early next week.
Have a good weekend.

0 Votes 0 ·

1 Answer

YanivLavi-3333 avatar image
1 Vote"
YanivLavi-3333 answered YanivLavi-3333 commented

For alerts to resolve we need to be able to know that the condition is no longer met.
The query generates a sparse time series. You will need to use make_series to provide a default value and handle the null values, so that the alert can resolve.

· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I have tried using make-series like below

 requests
 | where success == 'False' and customDimensions['API Name'] == 'my-api' 
 | make-series AggregatedValue = sum(itemCount) default=0 on timestamp step 10m

However I cannot save the condition due to this error message Search Query should contain 'AggregatedValue' and 'bin(timestamp, [roundTo])' for Metric alert type. Please note that I'm using Metric measurement.

Basically I'm interested in alerting if in the last 10 minutes we had more than 100 failed requests for a given API in our API Management resource.

0 Votes 0 ·

I have simplified the query/alert to use Number of results instead of Metric measurement as per attachment and the alert now resolves, but it does so with a significant delay (as per attachment).

114195-screenshot-2021-07-13-at-130613.png
114169-screenshot-2021-07-13-at-130507.png


0 Votes 0 ·

You can still use measurement in this way:

requests
| where success == 'False' and customDimensions['API Name'] == 'my-api'
| make-series AggregatedValue = sum(itemCount) default=0 on timestamp step 10m
| summarize AggregatedValue = sum(AggregatedValue) by bin(timestamp,10m)

We removed the need to do this in the query in the new API preview.
This resolved logic is explained here and accounts for ingestion delays:
https://docs.microsoft.com/en-us/azure/azure-monitor/alerts/alerts-unified-log#state-and-resolving-alerts

0 Votes 0 ·

The query you have provided fails with bin(): argument #1 - invalid data type: dynamic

I made it "compile" using mv-expand:

requests
| where success == 'False' and customDimensions['API Name'] == 'my-api'
| make-series AggregatedValue = sum(itemCount) default=0 on timestamp step 10m
| mv-expand timestamp to typeof(datetime), AggregatedValue to typeof(real)
| summarize AggregatedValue = sum(AggregatedValue) by bin_at(timestamp, 10m, now())

However, the alert is not auto-resolving. Is there a way to diagnose each evaluation of the condition and check what the result was?

0 Votes 0 ·
Show more comments