In our production environment, when the system is under heavy load, one of our applications (let's call it "App X", a WCF service using .NET Framework 4.6.1) hosted by IIS is hitting a cap of "64" for ASP.NET Apps v4.0.30319 Requests Executing. We've not made any configuration changes to the site (everything should be default), and it's the only site in the pipeline that seems to hit this cap.
All requests that come to this app first go through a different WCF app (let's call this one "App Y") that may be processing 1,000+ requests at a time (so, clearly not hitting the same 64 request cap as App X). I'm guessing these requests are piling up because App X is stuck at only processing 64 requests at a time.
App X and App Y both have the same IIS configurations. We've tried the following:
- Reviewed all application pool and IIS site settings for any red flags or differences between App X and App Y
- Created a new IIS site and application pool to host App X
- Setting ServicePointManager.DefaultConnectionLimit = int.MaxValue; in the service's class constructor.
- Explicitly setting the maxConcurrentReqeustsPerCPU value in the aspnet.config file for .NET v4.0 (this actually caused the IIS site to stop working)
- Had a network expert monitoring performance and limits while the system was under heavy load in an effort to find red flags
- Confirmed there are no traffic limits on applicable ports / channels
All other settings that I've read about seemed to only be applicable for older versions of IIS or the .NET Framework.
The server hosting these sites run on Windows Server 2012 R2 Datacenter, with 6 logical processors.
Here's are two screen caps of the performance monitors while the system is under heavy load. This first one is of App Y, which is the first WCF application to receive the requests in this pipeline:
This next one is of App X, which is the WCF application that seems to only process 64 requests at a time:
We are not sure why this cap of 64 requests is being hit. The CPU and memory on all servers (including our database server) is well within reasonable limits.
Any guidance or direction on things we should try would be a huge help.