Windows Server 2003, is your network bandwidth all going to 1 CPU? Here is how to fix it.
Windows Server 2003 Service Pack 1 + KB 912222
The Microsoft Windows Server 2003 Scalable Networking Pack release
Windows Server 2003 Service Pack 2
Note: Mike Lagase has a screen shot of what task manager looks like here Processor 0 increased CPU utilization
Back in the Windows NT 4.0 Server, Windows 2000 Server, Windows Server 2003 RTM/Service Pack 1, we had to use this registry key to load balance the incoming network i/o between the processors:
When we launched Service Pack 2 for Windows Server 2003, there was this new feature called “Receive Side Scaling” (RSS) that would let the network traffic being received be dispersed amongst the different CPU’s. It was a part of the “Scalable Network Pack” (SNP).
What is Receive Side Scaling (RSS)?
Leverage multiple processors for greater parallelism in network receives.
Good for heavy receive processing with large numbers of connections.
i.e. SQL servers, Exchange servers, File servers, Sharepoint servers, Web servers, FTP servers, etc…
It was first available as a hotfix KB 912222 in Windows Server 2003 Service Pack 1. Which it was included with “Service Pack 2”.
For us and you the administrator, we saw a spike on network related issues. Which are documented in these blogs and articles:
- Common Networking Issues After Applying Windows Server 2003 SP2 on SBS
- 936594 You may experience network-related problems after you install Windows Server 2003 SP2 or the Scalable Networking Pack on a Windows Server 2003-based computer
- 948496 Update to turn off SNP features for Windows Server 2003 and Windows SBS 2003
We went ahead and pushed out KB 948496 via Windows Updates, one of those rare non-security hotfixes that are hosted there.
What we found afterwards, most of the issues were due to TCP Chimney and not RSS or TCPA.
So what is the recommendation today (as of at least Feb. 2009) for RSS?
WARNING: RSS is incompatible with NAT or with Network Load Balancing (NLB), so for those, please leave it turned off and use KB 892100
WARNING 2: Test, test, and test.
1) First, check in Task Manager, to see if you observe the symptom (i.e. Do you have high cpu utilization on processor 0 due to high incoming network load?).
Note: You might want to take a perfmon to see if certain scheduled jobs (i.e. Backups) make the network load and processor 0 utilization go up.
If no, nothing needs to be done.
If yes, continue to step 2 below.
950224 A Scalable Networking Pack (SNP) hotfix rollup package is available for Windows Server 2003
Note: You need to make sure that Step 3 is taken care of when you install this hotfix, since it’s a fix from us (Microsoft) and your NIC vendor.
3) Update the NIC driver with a version newer than Feb. 2009.
Note: You need to make sure that Step 2 is taken care of, since it’s a fix from us (Microsoft) and your NIC vendor.
Note 2: I would personally update to one that is at least within a year from today. Search the hardware vendors newsgroups for issues with different versions of the NIC driver.
4) Disable TCP Offload
5) Re-enable RSS
EnableRSS (DWORD) 1 (hex)
By the time you are done, you should see:
- All processors can be used simultaneously for incoming network traffic.
- Incoming connections distributed across available processors.
Windows Scalable Networking Initiative
Receive-Side Scaling Enhancements in Windows Server 2008
Setting the Number of RSS Processors (Windows Drivers)
List of Network related hotfixes post Service Pack 2 for Windows Server 2003 SP2