We ran the below PowerShell cmdlet and found that there are two resources which are offline in cluster database but not present in Failover Cluster Manager (FCM) console.
Get-ClusterResource | where {$_.state -eq "Offline"}
We discussed this with SQL DBA and he confirmed that these resources were removed as part of a maintenance activity last weekend
Recommendation:
If you are decommissioning any cluster node in future make sure you remove them from SCOM and then decommission them. Anyway, it is a best practice to always remove the SCOM Agent before decommissioning any server or else it will leave the entries in SCOM and can create various problems and inconsistencies.
The orphaned resource issue is a very edge case scenario. And you might not run into later. However, just to be 100% sure we will not run into this issue, make sure after any maintenance in cluster, the output of the below PowerShell cmdlet is NULL.
Get-ClusterResource | where {$_.state -eq "Offline"}
** Bear in mind that there might be cluster resource which can be offline. That is absolutely fine. But if you find some resource offline in PowerShell but the same resource is not present in Failover Cluster Manager then those are pretty much orphan. The cluster admin can confirm that.
We removed both the orphan resources using the below PowerShell cmdlets.
Remove-ClusterResource -Name "File Server (\XXXXXXX)"
Remove-ClusterResource -Name "SQL Network Name (\YYYYYY)"
Restarted the Health Service on all 3 nodes. After that all the cluster objects were discovered in SCOM. Also the missing SQL Instances were discovered.
This a by design behavior of the SCOM Cluster Discovery that whenever it detects orphan resources in the cluster it will remove the cluster from SCOM. This is documented in the below article.
Support Tip: Some or all Cluster resources are not discovered by the OpsMgr agent - Microsoft Tech Community
https://techcommunity.microsoft.com/t5/system-center-blog/support-tip-some-or-all-cluster-resources-are-not-discovered-by/ba-p/349391