I set up an alert in AKS to detect when pod count is below 1. I use AZ portal. However the alert is never triggered.
I believe the signal is correct, as I can see the blue line going up and down when I modify the number of pods I wanna track. And I can see the threshold line is in between. So I cannot figure out why the signal would be wrong. I have tried different aggregations, by average, by total, still no trigger.
I also followed this: https://docs.microsoft.com/en-us/azure/azure-monitor/alerts/alerts-troubleshoot
Still, alert never triggered.
Any ideas on how to troubleshoot this issue?
More info: https://github.com/Azure/AKS/issues/2183