kb5009546 seems to break SCOM console

Question

I look after quire a large SCOM (2019 RU3)environment, recently the management console of 2 of 7 management servers are unable to open the management console (after install of update (kb5009546)) with the message
“Application: Operations Manager
Application Version: 10.19.10505.0
Severity: Error
Message: Failed to connect to server 'mgrscom3.domain.com’
Microsoft.EnterpriseManagement.Common.ServiceNotRunningException: The Data Access service is either not running or not yet initialized. Check the event log for more information. ---> System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://'mgrscom3.domain.com':5724/DispatcherService. The connection attempt lasted for a time span of 00:00:02.0051592. TCP error code 10061: No connection could be made because the target machine actively refused it 10.192.0.1:5724. ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.192.0.1:5724
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)

and some more info very similar to other posts I have seen , I have tried to follow the advice and nothing seems to work so far.
Trying
$testport = new-object System.Net.Sockets.tcpclient("'mgrscom3.domain.com","5724")
$testport.Connected
I get
new-object : Exception calling ".ctor" with "2" argument(s): "No connection could be made because the target machine actively refused it 10.192.0.1:5724"
At line:1 char:13

• $testport = new-object System.Net.Sockets.tcpclient("mgrscom3.

Same test from other server (to their own name) work
Same test to server with issues from other servers give the same error.
Why is port 5724 not listening
There is no firewall rules in between servers , windows fire wall is disabled.
There are a few events that repeat like ->
-System Center Data Access Service and/or System Center Management is unresponsive because Authorization Manager is unable to recover from database errors. Please restart services System Center Data Access Service and System Center Management.
And
The System Center Data Access service failed due to an unhandled exception.
The service will attempt to restart.
Exception:

Microsoft.EnterpriseManagement.Common.SdkServiceNotInitializedException: The Data Access service has not yet initialized. Please try again.
at Microsoft.EnterpriseManagement.ServiceDataLayer.DispatcherService.get_Container()
at Microsoft.EnterpriseManagement.Mom.Sdk.Service.SdkSubService.SdkChannel.Start()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()

if I point the management console to another server it works, if I try to point another console to the troubled server it also fails.

To me the issue is port 5724

Tried netstat on bad server and can see 5723 everywhere, but no 5724

On other good servers I can see stuff like
TCP [::1]:5724 mgrscom3.domain.com:51269 ESTABLISHED
TCP [::1]:5724 mgrscom3.domain.com:55008 ESTABLISHED
TCP [::1]:5724 mgrscom3.domain.com:55400 ESTABLISHED
TCP [::1]:5724 mgrscom3.domain.com:55403 ESTABLISHED
TCP [::1]:5724 mgrscom3.domain.com:57823 ESTABLISHED
TCP [::1]:51269 mgrscom3.domain.com:5724 ESTABLISHED
TCP [::1]:55008 mgrscom3.domain.com:5724 ESTABLISHED
TCP [::1]:55400 mgrscom3.domain.com:5724 ESTABLISHED
TCP [::1]:55403 mgrscom3.domain.com:5724 ESTABLISHED
TCP [::1]:57823 mgrscom3.domain.com:5724 ESTABLISHED

I have looked through numerous articles and tried some suggestions but the issue persists.

Uninstalling the patch does not fox the issue for me

Answer 1

Interesting. I would definitely open a support case with Microsoft if you have that capability. Make sure the core services are running. Restart these services and check the Application and Operations Manager event logs for more telling events. Check to see if there are any blocks in Windows Firewall and your host-based IDS service. Nothing really stands out in the KB documentation. Since this is a newly release patch it should be clear soon if others are having the same issue.

Answer 2

I don't know the root cause or if that KB is indeed involved, but I can tell you this is SCOM Server issue, not a console issue. More specifically a Data Access service issue.
Said otherwise, your SCOM server is broken and that's why the console can't connect.

Answer 3

I put the word out to a few other SCOM admins. Waiting to see if others report the same. I don't have a lab up at the moment to test. It would seem if the KB is the cause, that it is interfering with 5724 or the authentication or the OMSDK service operation. I assume the core services are running rather than crashing? Have you tried reinstalling the console?

Answer 4

Are you able to do an UDL test from the problematic management server to database server?

I would be interested to know about the drivers installed on the problematic management server and TLS settings between the management server and database server.

Answer 5

This patch is for Windows 10 and Server 2016. A fellow SCOM admin installed the same patch on Server 2016 with SCOM 2019 UR3 with no negative results. It seems maybe an IDS type issue.