Diagnosing Web Performance Issue

Question

I have a ASP.NET web server application (Framework 4.7.1) that I'm trying to understand why performance is limited in specific scenarios and what I can do to optimize or increase it. It is written to be async all the way down.

I am attempting to execute as many requests per second using WebSurge to call Web API service method that takes a specified number of milliseconds on the query string which simply executes an async Task.Delay for that amount of time, and then navigate through different pages of the application.

When this delay is small, e.g. 50 ms or below and hundreds of requests per second are being executed for this Web API call via WebSurge, the performance is excellent, other quick web service requests on different pages held up, and pages don't take long at all to load up. However, as I increase this delay to two seconds or more and have 32 threads attempting to hit the server and respond as fast as it can, the performance starts to slow to a crawl.

As I said, everything in the website should be async all the way down. What is the performance bottleneck here?

I’m not sure what additional information is required. The CPU usage is low and all I’m doing is navigating through the same few three or four different pages as a single user that calls some Web APIs that retrieve data from the database that are all normally very fast under no load (under 30 ms), in addition to this is executing in another performance testing app (WebSurge) a singular API call hundreds of times or however many times it can, that is just an async Task.Delay for a specified time. So as I increase this time from 10 ms to 2 s and hit it with WebSurge increasing the threads from 8 to 16 to 32 with minimal time between request calls, as soon as I hit 32 users and a 2s async delay time, the navigation of one user between pages starts to become slow. I can look at the underlying pending API calls for the pages where these requests run pretty quick (10, 30, 50 ms), but many of them just sit in pending state for several seconds. So what I’m doing is simulating a server load trying to figure out why it slows to a crawl with so few users. The CPU usage is low so a lot of async processing is growing on, but I would expect those long running async task calls not to impact the processing of the web pages that have much quicker requests happening behind the scenes.

Answer 1

I was informed by someone that it's likely due to the Windows 10 Pro connection limit of 10 for app pools on IIS.

So it's the limited connections for the app pool requests that are waiting on a response for the async calls and not the async calls themselves which have already been quickly handled on the CLR thread pool. That would make sense as to why none of the changes that I made to machine.config or Aspet.config in the appropriate framework directories, or even changing the ServicePointManager.DefaultConnectionLimit to int.Max did nothing (which would change the connection limit of 10 if it was possible).

The latter would only work on something like Windows Server I presume. I believe this to be the case but I'm not certain.

Answer 2

I would recommend running bench mark scenarios specifically in your unit tests their no sure fire way to understand the bottle necks.

If its data related and an old system using entiyframework for reads its probably that is ur bottle neck.

Ef 6 and Ef core have made fast improvements in that area if you can upgrade I would other wise setup some unit tests and install this nuget.

https://github.com/dotnet/BenchmarkDotNet

Its one of the many bench marking tools available to us as developers. You would spot this performance easy if u have unit tests testing your business logic.

BTW Your understanding of async is all wrong just cause it is async doesn't mean it will be fast async is just a method signature that doesn't block the ui.

When you say some sections of the web api are fast some are slow you have not detailed are the slow areas fetching a million rows or more. Not that an api ever should it always should be well under that.

Answer 3

I have had no issues with signular calls in that case I would suggest the table may not have an index or proper key attached to it.

Async is only used not to block the main threads but if your implementation of it is wrong then thats where the issue is.

Please post an mvp so we can understand your calls obv if company source code hide certain tables name etc.