question

DanielA-3857 avatar image
0 Votes"
DanielA-3857 asked DanielA-8699 answered

SFB2019: Get-CsPoolFabricState shows Health: Warning but everything appears ok

Hi All,

When running Get-CsPoolFabricState for the Skype for Business 2019 pool i have down the bottom of the output it has:

 *Pool All Server and Services Summary:
 Fqdn: SKYPE19-FE-BD1.EXMAPLE.COM Primary: 34 Secondary: 70
 Fqdn: SKYPE19-FE-BD2.EXMAPLE.COM Primary: 35 Secondary: 69
 Fqdn: SKYPE19-FE-BD3.EXMAPLE.COM Primary: 35 Secondary: 69
 WARNING: Fqdn: SKYPE19-FE-BD1.EXMAPLE.COM - Health: Warning Status: Up [Seed Node] Primary: 3 Secondary: 0
 WARNING: Fqdn: SKYPE19-FE-BD2.EXMAPLE.COM - Health: Warning Status: Up [Seed Node] Primary: 0 Secondary: 3
 WARNING: Fqdn: SKYPE19-FE-BD3.EXMAPLE.COM - Health: Warning Status: Up [Seed Node] Primary: 0 Secondary: 3
 WARNING: One or more servers are shutdown, unhealthy or deactivated.  Ensure they are running and activated.  Restart the server if problems persist.*

Has anyone seen this before? I cannot find any events in the event log which have a warning (Lync Server log is clean). Everything with pool appears to be working correctly so I have no idea what it thinks is warning.

I have run

Reset-CsPoolRegistrarState -ResetType:"FullReset"

this completed fine but the warning still persisted (its been 8 hours since running fullreset).



office-skype-business-server-administration
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

You're advised to post your output in "code" block so the system will not treat your post as spam (as the domain names are automatically converted to links). I've edited your post to do that for you.

1 Vote 1 ·
DanielA-3857 avatar image
2 Votes"
DanielA-3857 answered SharonZhao-MSFT commented

So I finally figured out what this was. In my case it was caused by my Server Default certificate that was due to expire within 90 days (it was still valid).

Here is how I debugged it (this stuff is not documented anywhere from what I can find)

From powershell on one of the front end nodes run:

 PS> Connect-ServiceFabricCluster
    
 PS> Get-ServiceFabricClusterHealth
    
    
 AggregatedHealthState   : Warning
 UnhealthyEvaluations    :
                           Unhealthy nodes: 100% (3/3), MaxPercentUnhealthyNodes=0%.
    
                           Unhealthy node: NodeName='SKYPE19-FE-BD2.EXMAPLE.COM', AggregatedHealthState='Warning'.
    
                                 Unhealthy event: SourceId='System.FabricNode', Property='Certificate_cluster', HealthState='Warning', ConsiderWarningAsError=false.
    
                           Unhealthy node: NodeName='SKYPE19-FE-BD3.EXMAPLE.COM', AggregatedHealthState='Warning'.
    
                                 Unhealthy event: SourceId='System.FabricNode', Property='Certificate_cluster', HealthState='Warning', ConsiderWarningAsError=false.
    
                           Unhealthy node: NodeName='SKYPE19-FE-BD1.EXMAPLE.COM', AggregatedHealthState='Warning'.
    
                                 Unhealthy event: SourceId='System.FabricNode', Property='Certificate_cluster', HealthState='Warning', ConsiderWarningAsError=false.
    
 NodeHealthStates        :
                           NodeName              : SKYPE19-FE-BD2.EXMAPLE.COM
                           AggregatedHealthState : Warning
    
                           NodeName              : SKYPE19-FE-BD3.EXMAPLE.COM
                           AggregatedHealthState : Warning
    
                           NodeName              : SKYPE19-FE-BD1.EXMAPLE.COM
                           AggregatedHealthState : Warning
    
 ApplicationHealthStates :
                           ApplicationName       :
                           AggregatedHealthState : Ok
    
                           ApplicationName       : fabric:/System
                           AggregatedHealthState : Ok
    
 HealthEvents            : None


To drill down into error run (you can target any of the nodes in the above output):

 PS> Get-ServiceFabricNodeHealth -NodeName "SKYPE19-FE-BD2.EXMAPLE.COM"
    
    
 NodeName              : SKYPE19-FE-BD2.EXMAPLE.COM
 AggregatedHealthState : Warning
 UnhealthyEvaluations  :
                         Unhealthy event: SourceId='System.FabricNode', Property='Certificate_cluster', HealthState='Warning', ConsiderWarningAsError=false.
    
 HealthEvents          :
                         SourceId              : System.FabricNode
                         Property              : Certificate_cluster
                         HealthState           : Warning
                         SequenceNumber        : 132555760970446011
                         SentAt                : 20/01/2021 12:28:17 AM
                         ReceivedAt            : 20/01/2021 12:30:17 AM
                         TTL                   : Infinite
                         Description           : Certificate expiration: (2021-04-15 06:02:00.000, 81f707cef7d097bc1a0db3c32c213486f501129e)
                         RemoveWhenExpired     : False
                         IsExpired             : False
                         Transitions           : Ok->Warning = 20/01/2021 12:30:17 AM, LastError = 1/01/0001 12:00:00 AM
    
                         SourceId              : System.FM
                         Property              : State
                         HealthState           : Ok
                         SequenceNumber        : 2
                         SentAt                : 20/01/2021 12:29:13 AM
                         ReceivedAt            : 20/01/2021 12:30:46 AM
                         TTL                   : Infinite
                         Description           : Fabric node is up.
                         RemoveWhenExpired     : False
                         IsExpired             : False
                         Transitions           : Warning->Ok = 20/01/2021 12:30:46 AM, LastError = 1/01/0001 12:00:00 AM


Simply replacing the certificate using https://docs.microsoft.com/en-us/powershell/module/skype/set-cscertificate?view=skype-ps ( or the gui if thats your thing) caused the warnings to clear-up.

Hopefully this helps someone else.



· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@DanielA-3857
Thanks for your sharing. I'm glad that you found out the root cause of this issue and fixed it.
Any other questions, please feel free to discuss with us here.
Have a nice day!

0 Votes 0 ·
SharonZhao-MSFT avatar image
0 Votes"
SharonZhao-MSFT answered DanielA-3857 commented

@DanielA-3857,

Do you do any changes on your environment recently?

You could check if all the servers in your environment run well in Skype for Business Server Control Panel as below:
58572-image.png

For reference, you could read this article to troubleshoot front end service cannot be started.


If the response is helpful, please click "Accept Answer" and upvote it.

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.





image.png (69.8 KiB)
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

No changes. But recently had a certificate expire. The certificate is now fixed. Everything in the control panel is green

0 Votes 0 ·
DanielA-3857 avatar image
0 Votes"
DanielA-3857 answered

No changes. But recently had a certificate expire. The certificate is now fixed. Everything in the control panel is green

58835-image.png



image.png (55.5 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

DanielA-8699 avatar image
0 Votes"
DanielA-8699 answered

You can also clear these alerts by running the following for each node:

 Send-ServiceFabricNodeHealthReport -NodeName:'SKYPE19-FE-BD2.EXMAPLE.COM' -SourceId:'System.FabricNode' -HealthProperty:'Certificate_cluster' -HealthState Ok -Description "Upcoming cert expiry accepted"
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.