question

RandallBreneman-6024 avatar image
0 Votes"
RandallBreneman-6024 asked RandallBreneman-6024 edited

How to scale out App Service Plan based on HTTP status 503 errors?

When our App Service Plan is overloaded, many clients get HTTP status 503: Service Unavailable errors. I haven't been able to find any data in available metrics that correlates closely enough to 503 errors to effectively react with scale out. One part of the service's client base uploads error logs to the service that are persisted in the database. Enough of them are getting through during these periods that this data would be an appropriate scale out factor if I could figure out how to implement it. I looked at Logic Apps that can query for the persisted error logs but I can't find any way for a Logic App to implement App Service Plan scale out function. I am far from being an expert in all Azure services. Can you give me any ideas to implement scale out in this scenario? Thank you.

Update: I observed a spate of 503's one night lasting from 00:00 UTC through 07:00 UTC. These are off-peak hours and all resource demands were low. There was no maintenance of any kind on our behalf occurring. This spate of 503's was a singular, random occurrence and follows no observable pattern in any resource metric.
Conclusion: 503's were possibly a result of some transient, unknown issue with Azure infrastructure. As long as it doesn't become common, scaling out for 503's is probably not a productive use of time. Whether or not scaling out would have done any good in this scenario is unknown.

azure-webappsazure-webapps-scaling
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

VidyaNarasimhan-2409 avatar image
1 Vote"
VidyaNarasimhan-2409 answered RandallBreneman-6024 commented

Hi @RandallBreneman-6024,

The best practice is to configure auto-scale for App Service based on a threshold e.g. 70%-80% CPU/Memory utilization before the App Service Plan is overloaded. You can arrive at a threshold by monitoring the App Service plan,

But to answer you question on scale out using custom metrics like request/service failure, below are few options:

One way to scale out using custom metrics is by enabling Application Insights for the App Service as described here https://docs.microsoft.com/en-us/azure/azure-monitor/autoscale/autoscale-custom-metric
Application Insights also has standard metrics that include count of failed requests https://docs.microsoft.com/en-us/azure/azure-monitor/app/standard-metrics#failure-metrics

To scale out using Azure Functions, you can try to write a powershell function as per this link https://docs.microsoft.com/en-us/azure/app-service/scripts/powershell-scale-manual


· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thanks for the hints. None of the scale out metrics like CPU peak in any way that is closely correlated with the 503 errors. I've seen some data-in peaks in about the same time frame but the peaks are weakly correlated and would still miss most of the 503 error periods. The other two suggestions look like good ones. I hadn't considered application insights for a service that is only for data exchange API before, but I'm not experienced with it. The links are much appreciated, thank you.

0 Votes 0 ·
AliSufyan-1625 avatar image
0 Votes"
AliSufyan-1625 answered

Azure App Service environments support autoscaling. You can autoscale individual worker pools based on metrics or schedule. Autoscaling optimizes your resource utilization by automatically growing and shrinking an App Service environment to fit your budget and or load profile.

You scale up by changing the pricing tier of the App Service plan that your app belongs to. Scale out: Increase the number of VM instances that run your app. You can scale out to as many as 30 instances, depending on your pricing tier.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.