question

JohnKay-1160 avatar image
0 Votes"
JohnKay-1160 asked LimitlessTechnology-2700 answered

Cluster High CPU usage S2D FSLogix storage

We have a two node Server 2019 cluster running the Scale Out File Server role for Storage Spaces direct. We use this to host our FSLogix Profiles for Remote Desktop Services. The cluster runs nothing else and is on the latest updates.

After about 30 users are signed into RDS the cluster CPU seems to spike the System service and will spike to 100% and sometimes stay up there for a while. We've disabled all Antivirus, uninstalled Defender firewall, updated all drivers and firmware, and rebooted multiple times. When the issue first started to occur we thought it might be our disk IO so we put in double the disk and went from 6Gb Sata intel DC SSD drives to 12Gb SAS SSD drives with way better performance. Issue still occurs. The strange thing is we had an error about the max envelope size being hit so we increased it in the registry, this fixed our issue for about 10 days, then suddenly it came back. This time we don't see any errors like we did before about the max envelope size being exceeded. The cpu in each server is a Intel E5-2670V3 2.3 ghz 12 core CPU. Since this is only a file server I would assume this would be plenty. Before this all started happening we would have about 85 users signed into RDS at once with 1-3% CPU usage on the cluster. Now with about 30 users the average CPU is around 30% with spikes to 100% and sometimes holding there.

Any suggestions?

windows-server-clusteringwindows-server-fslogix
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

JohnKay-1160 avatar image
0 Votes"
JohnKay-1160 answered

I am seeing this in the event viewer, not sure what this means though.

The maximum file size for session "EventLog-Microsoft-Windows-SMBServer-Analytic" has been reached. As a result, events might be lost (not logged) to file "C:\Windows\System32\Winevt\Logs\Microsoft-Windows-SMBServer0x8800081Analytic.etl". The maximum files size is currently set to 1048576 bytes.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

JohnKay-1160 avatar image
0 Votes"
JohnKay-1160 answered

We removed the E5-2670V3 CPU from one of the nodes, then replaced the single CPU with 2x E5-2680V3 and moved the role over to it. Same issues, and it has maxed the CPU out on it as well.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

JohnKay-1160 avatar image
0 Votes"
JohnKay-1160 answered

We opened CMD and ran netstat -abo and received this.  TCP   [fe80::796b:41a:4ab5:d1ef%11]:445 StorageNode02:51278   ESTABLISHED   4 Can not obtain ownership information  TCP   [fe80::796b:41a:4ab5:d1ef%11]:445 StorageNode02:51955   ESTABLISHED   4 Can not obtain ownership information  TCP   [fe80::796b:41a:4ab5:d1ef%11]:49696 StorageNode02:ms-cluster-net ESTABLISHED   5080 [clussvc.exe]  TCP   [fe80::796b:41a:4ab5:d1ef%11]:49699 StorageNode02:microsoft-ds ESTABLISHED   4 Can not obtain ownership information  TCP   [fe80::796b:41a:4ab5:d1ef%11]:49700 StorageNode02:microsoft-ds ESTABLISHED   4 Can not obtain ownership informationThe PID is 4 which is the service that is maxing our CPU out.Any suggestions?

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

LimitlessTechnology-2700 avatar image
0 Votes"
LimitlessTechnology-2700 answered

Hello,

  1. Please use process explorer and procmon to monitor which process is being consumed high CPU.

  2. Please check what kind of applications are used 30 users, are they developers or just taking rdp for Office applications.

  3. Please use disk iops tool to monitor read/write on disks.

  4. I believe CPU spikes for momentary is acceptable in RDS environment as they logon and logoff frequently which loads and unloads their user profile in to RAM.

  5. Please check below Microsoft article for further tuning for RDS environment.


https://docs.microsoft.com/en-us/windows-server/administration/performance-tuning/role/remote-desktop/session-hosts



If the reply was helpful, please don’t forget to upvote or accept as answer.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.