question

kvancleave-9880 avatar image
0 Votes"
kvancleave-9880 asked GitaraniSharmaMSFT-4262 commented

One of our Azure WAF Instances suddenly starts running at 100% CPU

We are running right now with the WAFs set at a minimum of 10, which is way more than we need in an attempt to avoid this issue, but it keeps happening.

We will be running fine, and then suddenly one of the WAF units out of the 10 we normally run starts running at 100% CPU while all the others are not running over 25% CPU. We cannot see what the individual WAFs are doing in the Azure portal, but Microsoft looked deeper into the issue and told us this is what was happening, but they do not know why.

This causes our entire website to start running very slow and it continues until we manually add or remove a WAF unit, which seems to reset something. This is a trick a Rackspace tech told us about to make the problem stop.

Rackspace has engaged Microsoft on this issue multiple times, but no one can tell us why this happens or what they can do to prevent it. Honestly, it sounds like a WAF software issue in my opinion.

We have redeployed the WAFs several times so far in an attempt to get on better hardware, but sooner or later this same issue comes up again and it really makes our customers scream for several minutes until we notice the issue and can add or remove a WAF instance.

This happened again today. We really need a good solid answer and solution to this problem. We are a telemedicine company so when we have issues because of this, it is much more serious than an average website.

azure-web-application-firewall
· 8
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @kvancleave-9880 ,

Apologies for the delay in response.

Could you please let us the below details?

1) Is this WAF enabled on Application gateway?
2) What is the SKU - v1 or v2?
3) If v2, do you have autoscaling enabled?

Regards,
Gita

0 Votes 0 ·

Here are the answers to your questions:

1) no
2) V2
3) Yes

0 Votes 0 ·

Thank you for the answers, @kvancleave-9880 but I'm a bit confused when you say that WAF is not enabled on Application gateway and the SKU is V2 with autoscaling enabled.
Could you please validate that the WAF is enabled on an Application gateway or an Azure Front Door?

0 Votes 0 ·

Cloudflare is our WAF. The WAF on the Application Gateway is actually enabled however, it is in detection mode only.

0 Votes 0 ·

Thank you for the update, @kvancleave-9880.

But you have mentioned that "One of your Azure WAF Instances suddenly starts running at 100% CPU".
I would need some details on your setup to get more clarity on the issue. The WAF running at 100% CPU is Cloudflare or Azure WAF?

0 Votes 0 ·

They are in Azure. We have Azure typically set to run a minimum of 10 WAFs so the 100% CPU issue does not completely take us down. A MS tech got involved at one point and told us that one of them suddenly went to 100% for no readon they can see. We do not have visiblity to this kind of detail in our Azure portal. We were told that we can change the minimum either up or down by 1 and that will reset things and fix the problem. It does fix the problem, but not before we are getting tons of complaints from the customers about the system running super slow.

0 Votes 0 ·

Hello @kvancleave-9880 ,

If I understand correctly, you have Cloudflare WAF deployed in Azure and this is causing 100% CPU issue. But to get further clarity, I would request you to send us an email with the support ticket numbers that you have raised in the past with Microsoft support team and we will look into them to find out the exact problem.

Please send us an email as advised in the private message.

Regards,
Gita

0 Votes 0 ·

Hello @kvancleave-9880 ,

Could you please provide an update on this post?

Please send us an email as advised in the private message, with the support ticket numbers that you have raised in the past with Microsoft support team and we will look into them to find out the exact problem.

Regards,
Gita

0 Votes 0 ·

0 Answers