question

danielbecroft avatar image
danielbecroft asked ·

Debugging 403 Ip Forbidden errors

We have some Azure WebApps (AppServices) deployed, and have implemented the IP address restrictions so that only traffic from a defined set of networks is allowed through.

Periodically, however, we are receiving HTTP 403 Ip Forbidden error messages on the AppService, even when access was previously allowed through, and neither the firewall rules, or our outgoing IP address, has changed.

It seems to come and go (we had one that was failing yesterday, but was working this morning). Deleting and recreating the webapp seems to resolve the issue, as well as removing all the IP address restrictions.

Is there a way to debug these errors? I can't see the responses in the Log Stream or any other item in the AppService blade, so it would be good to find somewhere that says "Request from IP x.x.x.x was blocked due to restrictions" or similar.

 > curl -I https://xxxxxx-xxxxxx-xxxxxx-315-xxxxxx-api.azurewebsites.net/api/Clients?status=Active
 HTTP/1.1 403 Ip Forbidden
 Content-Length: 2345
 Content-Type: text/html
 Server: Microsoft-IIS/10.0
 Date: Fri, 07 Feb 2020 04:27:00 GMT

The apps are deployed as Linux containers, running on a B1 service plan.

I have seen reference that this error could come about due to quotas being exceeded, but I can't see anything that refers to an exceeded quota (everthing appears to be within the allowed limits).

azure-webappsazure-webapps-securityazure-webapps-monitoring
3 comments
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thanks @SuwatBodin-7965 , I've verified that the external IP being detected by Azure is the correct one, and that it is correctly added in the firewall rules.

0 Votes 0 · ·

Try to see if this helps in checking to see if the IP is blocked due to IP restrictions. If the request is being blocked by IP restrictions it occurs on the FrontEnds of the infrastructure so the 403s would not be seen in the application logs. There's a detector available in the Azure Portal to confirm what IPs are being blocked if that is the cause, see below.

The access restrictions capability is implemented in the App Service front-end roles, which are upstream of the worker hosts where your code runs. Therefore, access restrictions are effectively network ACLs.

https://docs.microsoft.com/en-us/azure/app-service/app-service-ip-restrictions

3271-2020-02-21-15h28-53.png




0 Votes 0 · ·

1 Answer

VenkatakrishnanDamodaran-1003 avatar image
VenkatakrishnanDamodaran-1003 answered ·

Are you try to enable Web Server Logging and check?

Click App Service Logs -> Click File System in Web server logging. Also enabled Failed request tracing.

After the above settings. you will be able to see the traffic in Kudu Console via Advanced Tools-> Log directory.

5 comments Share
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thanks. I've checked the logs, and there's nothing appearing. The problem is that Azure is blocking the request before it gets anywhere near my container. There's nothing in any of the log files (that I can find) where Azure is recording a HTTP 403 IP Forbidden response.

0 Votes 0 · ·

Hi daniel,

What kind of firewall you are using. Are you enabling IP restrictions via web.config or you have front-end WAF?. If you have WAF. I have faced several issues with WAF for false positive traffic. you have to enable Logging for firewall via Storage account. Click "Diagnostic settings" in the WAF service, Click Diagnostic Settings-> provide name and select necessary log details and provide storage account. After that the all the traffic going through firewall logged in the storage account as a json file for each hour. you will be able to investigate that json file for any firewall false positive access denied issues. let me know how it goes.

Thanks for your patience.

0 Votes 0 · ·

Daniel, Apologies for the delay. Just checking in to see if you have had a chance to see the previous post. Kindly share the requested information to better assist you.

0 Votes 0 · ·

Sorry @ajkuma-MSFT . I don't have a separate WAF resource in front of the web app.

The solution was provided in a comment by another poster: using the "Diagnose and Solve problems", there's an option to check configuration and it will show the masked IP addresses that are being blocked. This is sufficient for our requirement.s

0 Votes 0 · ·
Show more comments