question

ChandlerZwolle-4083 avatar image
0 Votes"
ChandlerZwolle-4083 asked ChandlerZwolle-4083 commented

IIS Site Hitting Cap of 64 for "Requests Executing" Performance Counter

Hello,

In our production environment, when the system is under heavy load, one of our applications (let's call it "App X", a WCF service using .NET Framework 4.6.1) hosted by IIS is hitting a cap of "64" for ASP.NET Apps v4.0.30319 Requests Executing. We've not made any configuration changes to the site (everything should be default), and it's the only site in the pipeline that seems to hit this cap.

All requests that come to this app first go through a different WCF app (let's call this one "App Y") that may be processing 1,000+ requests at a time (so, clearly not hitting the same 64 request cap as App X). I'm guessing these requests are piling up because App X is stuck at only processing 64 requests at a time.

App X and App Y both have the same IIS configurations. We've tried the following:
- Reviewed all application pool and IIS site settings for any red flags or differences between App X and App Y
- Created a new IIS site and application pool to host App X
- Setting ServicePointManager.DefaultConnectionLimit = int.MaxValue; in the service's class constructor.
- Explicitly setting the maxConcurrentReqeustsPerCPU value in the aspnet.config file for .NET v4.0 (this actually caused the IIS site to stop working)
- Had a network expert monitoring performance and limits while the system was under heavy load in an effort to find red flags
- Confirmed there are no traffic limits on applicable ports / channels

All other settings that I've read about seemed to only be applicable for older versions of IIS or the .NET Framework.

The server hosting these sites run on Windows Server 2012 R2 Datacenter, with 6 logical processors.

Here's are two screen caps of the performance monitors while the system is under heavy load. This first one is of App Y, which is the first WCF application to receive the requests in this pipeline:
195682-image.png


This next one is of App X, which is the WCF application that seems to only process 64 requests at a time:
195656-image.png



We are not sure why this cap of 64 requests is being hit. The CPU and memory on all servers (including our database server) is well within reasonable limits.

Any guidance or direction on things we should try would be a huge help.

Thanks,
Chandler

dotnet-csharpwindows-server-iiswindows-wcf
image.png (11.2 KiB)
image.png (10.6 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Bruce-SqlWork avatar image
0 Votes"
Bruce-SqlWork answered ChandlerZwolle-4083 commented
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank you, sir.

Unfortunately, I don't see that value being set anywhere in our solutions. Additionally, it looks like it's applicable to .NET Framework 4.7.1+, and we are using 4.6.1. It's also for the HttpClientHandler class, which is not used anywhere in our service. Our service is constructed using a System.ServiceModel.ChannelFactory, and by creating a new channel (factory.CreateChannel).

I am not a WCF expert by any means. This architecture was implemented before I joined the team and I've been helping maintain the code base for several years now. Please let me know if I'm missing something.

Thanks,
Chandler

0 Votes 0 ·
Bruce-SqlWork avatar image
0 Votes"
Bruce-SqlWork answered ChandlerZwolle-4083 commented

you really should upgrade the project to 4.8 4.6.1 support ends next week.

it has been years since using 4.6.1, but I think its:

System.Net.ServicePointManager.DefaultConnectionLimit = MAX_CONNECTIONS;

note: the default was 2, so I assume your code has bumped somewhere. the outgoing limit was due concerns about a single process saturating a server.




· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thanks, Bruce. I will add an action item to upgrade to 4.8.

Setting ServicePointManager.DefaultConnectionLimit = int.MaxValue; in the service's class constructor is one of the things we tried (we set it for both App X and App Y). That assignment was actually no where to be found, so we added it an in effort to resolve our issue. We did not notice a change in behavior. That was the most hopeful solution that I've read about, originating from this post in the Sept-12-2011 update:
https://docs.microsoft.com/en-us/archive/blogs/tmarq/asp-net-thread-usage-on-iis-7-5-iis-7-0-and-iis-6-0

I really appreciate your insight into this matter. Any other thoughts or suggestions would be greatly appreciated.

Thanks,
Chandler

0 Votes 0 ·