question

DanielIonita-3867 avatar image
0 Votes"
DanielIonita-3867 asked Crystal-MSFT commented

Create S2D Cluster fails with Error 25325 "An error occurred while performing the operation"

Hi guys,


Hoping someone could point me in the right direction here...


Got 4 nodes provisioned in our VMM 2019 (latest build) server. The nodes are Server 2019 Datacenter core up-to-date (Jan '21).

We're trying to use VMM to provision an S2D Cluster. The Job works flawless all the way till Validating the cluster without any warnings. On the "Create Cluster" step it fails and cleans up afterwards.

Permissio-wise:

  • The VMM RunAs is a full Domain Administrator

  • The AD computer objects for the nodes themselves are added to the OUs with full read and Create Computer objects selected

  • The Administrators local security group of each node have as members: the VMM service account AND the VMM computer object


Besides the Validation Report found on each node (which is spotless clean) and the Event Viewer in VMM which basically gives the same cryptic Error, there's nothing to go on here...

Are tehre any other logs I could pursue to trace this down?


Thanks a lot!

windows-server-clusteringmsc-virtual-machine-manager-hyper-vmsc-virtual-machine-manager-clustering
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@DanielIonita-3867, From your description, I know the S2D cluster is failed to create with error 25325. For the error description, it is general and can't find out the cause. Here, we suggest to go to the 4 nodes and check the logs like event log to see if thee's any finding: Here is a link related to troubleshooting S2D for the reference:
https://docs.microsoft.com/en-us/windows-server/storage/storage-spaces/troubleshooting-storage-spaces

Hope it can help.

0 Votes 0 ·
DanielIonita-3867 avatar image
0 Votes"
DanielIonita-3867 answered Crystal-MSFT commented

Coming back to this, we ended up creating the cluster manually, then it will appear in VMM to be managed... Worth mentioning that the VMM server is 2019 latest build, but sat on Win Server 2016 Standard. This created problems with RSAT tools being older and misbehaving on some management aspects of the 2019 cluster (for example the Storage Provider was not recognised and kept throwing errors (exact issue here https://social.technet.microsoft.com/Forums/systemcenter/en-US/76fd6d71-49f4-4749-8f13-f2e9e95dcb57/vmm-2019-cannot-add-s2d-storage-provider-windows-server-2019-hyperconverged-cluster?forum=virtualmachinemanager ).

After in-place upgrading the underlying OS to 2019, removed and re-added storage provider and it all went through.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@DanielIonita-3867 , Thanks for the update. I am glad that we find the cause. it can help others who have the same issue. I appreciate your sharing here.

Thanks for your time and have a nice day!

0 Votes 0 ·
DanielIonita-3867 avatar image
0 Votes"
DanielIonita-3867 answered

Hi Crystal! Thanks for your answer.

I have reviewed that link and many more so far without success. There are no Event Logs on any node individually that would suggest a problem during the creation of the cluster.

I have a feeling this is related directly to the process of creating the cluster remotely failing at the "New-Cluster" cmdlet, but logs are very unhelpful or nonexistent.

Is there a way to enable additional logging, either on VMM side or on the cluster nodes themselves to help catch the issue?

Logs I've analyzed so far are:

  • event viewer on VMM server and all nodes

  • C:\Windows\Cluster\Reports on all nodes (only validation reports are here, nothing about cluster creation process)

  • C:\ProgramData\VMMLogs

The VMM end-script for the Cluster creation reads:

 # Get Host Group 'MARGE'
 $HostGroup = Get-SCVMHostGroup -ID "c16f9ec8-4390-45ad-88b0-227d4d2f32ad"
 # Get RunAs Account 'IMPELLING\SCVMMADMIN'
 $AdminRunAsAccount = Get-SCRunAsAccount -ID "9fac398b-96e6-4ef6-a93f-7ab489f1ff2b"
    
 # Get Host 'marge04.impelling.gunit, marge03.impelling.gunit, marge02.impelling.gunit, marge01.impelling.gunit'
 $VMHosts = @()
 $VMHosts += Get-SCVMHost -ID "ef70b37b-17b6-4219-b0a3-a4b41c2a3807"
 $VMHosts += Get-SCVMHost -ID "f1b9abb7-27be-4ddb-89f6-c6fd80db3c6a"
 $VMHosts += Get-SCVMHost -ID "dee7cda7-585b-4d4d-9f25-e8daae6d5f93"
 $VMHosts += Get-SCVMHost -ID "fb55ecce-6402-4de2-a12b-f66beca60101"
    
 $StaticIPAddress = @("172.20.20.1")
    
 $DCBSettings = @()
 # Get Virtual Network Adapter 'vSMB2, vSMB1, vSMB2, vSMB1, vSMB2, vSMB1, vSMB2, vSMB1'
 $VirtualNetworkAdapters = @()
 $VirtualNetworkAdapters += Get-SCVirtualNetworkAdapter -ID "16f9003a-a8ee-4f88-a247-94cad6ca7e48"
 $VirtualNetworkAdapters += Get-SCVirtualNetworkAdapter -ID "9fed1603-79ca-4364-b6d1-2485b7785135"
 $VirtualNetworkAdapters += Get-SCVirtualNetworkAdapter -ID "d860302b-4791-42f4-a73a-b4c093ed98c5"
 $VirtualNetworkAdapters += Get-SCVirtualNetworkAdapter -ID "89a70237-998f-4b86-89df-014a23b6e6ae"
 $VirtualNetworkAdapters += Get-SCVirtualNetworkAdapter -ID "777af89b-3fc4-439a-8d80-606d8bb296b3"
 $VirtualNetworkAdapters += Get-SCVirtualNetworkAdapter -ID "6a6d96bf-2013-43c8-a4dc-bc18481763ef"
 $VirtualNetworkAdapters += Get-SCVirtualNetworkAdapter -ID "41f5563e-8f9a-4869-8bf5-77e870d90ae1"
 $VirtualNetworkAdapters += Get-SCVirtualNetworkAdapter -ID "0930de20-ce4b-4f60-be2d-217a93d32cdd"
    
 $DCBSettingsItem = New-SCDCBSettings -AddDCBSettings -SMBPriority 3 -SMBBandwidth 50 -ClusterHeartbeatPriority 7 -ClusterHeartbeatBandwidth 1 -VirtualNetworkAdapters $VirtualNetworkAdapters
 $DCBSettings += $DCBSettingsItem
    
 Install-SCVMHostCluster -ClusterName "MARGE" -JobGroup "ccc29df5-72b9-4387-bf21-8ce0579d6ff0" -RunAsynchronously -EnableS2D -Credential $AdminRunAsAccount -VMHost $VMHosts -ClusterIPAddress $StaticIPAddress -DCBSettings $DCBSettings

Appreciate the help!

Thanks!




5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

DanielIonita-3867 avatar image
0 Votes"
DanielIonita-3867 answered Crystal-MSFT commented

Hi again,

Coming back with more logs...

I went as far as enabling an ETL trace in VMM for debugging and identified the segment where it fails:

 [8]1AEC.1E8C::‎2021‎-‎01‎-‎15 09:40:42.238 [Microsoft-VirtualMachineManager-Debug]4,4,private\product\engine\ADHC\Operations\InstallClusterService.cs,98,InstallClusterService.InstallCluster [s#1 58496ms] Failing [ex#13b] New exception = [[(FaultException`1<ErrorInfo>#1abcba3) System.ServiceModel.FaultException`1[Microsoft.VirtualManager.Utils.ErrorInfo]: The creator of this fault did not specify a Reason. (Fault Detail is equal to CreateClusterFailed (25325)).]] (this: (InstallClusterService#[2]) { tbd }, clusterName: (string) "MARGE", nodes: (String[][3]#17c3e99) { "marge04.impelling.gunit", "marge03.impelling.gunit", "marge02.impelling.gunit" }, ignoredNetworks: (String[][2]#217de62) { "172.20.15.0/27", "172.20.17.0/27" }, staticIPs: (String[][0]#3f53e18) { }, addStorage: (Boolean#0) False),{00000000-0000-0000-0000-000000000000},
 [12]0D74.1E38::‎2021‎-‎01‎-‎15 09:40:42.282 [Microsoft-VirtualMachineManager-Debug]4,4,private\product\common\vmmhost\HostHelper\HostHelper.cs,306,Entering [s#27e4a82e] HostHelper`2.CloseProcess(this: (HostDuplexHelper`3<IInstallCluster, InstallClusterService, IClusterProgress>#[5]) {  }),{00000000-0000-0000-0000-000000000000},
 [12]0D74.1E38::‎2021‎-‎01‎-‎15 09:40:42.282 [Microsoft-VirtualMachineManager-Debug]4,4,private\product\common\vmmhost\HostHelper\HostHelper.cs,320,Checking remote interface for communication fault,{00000000-0000-0000-0000-000000000000},
 [12]0D74.1E38::‎2021‎-‎01‎-‎15 09:40:42.282 [Microsoft-VirtualMachineManager-Debug]4,4,private\product\common\vmmhost\HostHelper\HostHelper.cs,367,Process is still running,{00000000-0000-0000-0000-000000000000},
 [12]0D74.1E38::‎2021‎-‎01‎-‎15 09:40:42.282 [Microsoft-VirtualMachineManager-Debug]4,4,private\product\common\vmmhost\HostHelper\HostHelper.cs,374,Requesting process to exit,{00000000-0000-0000-0000-000000000000},

The only odd thing I find in this trace is the fact that staticIPs: (String[][0]#3f53e18) { } is empty, where it should have 172.20.20.1 as the cluster's IP address specified in the wizard (statically not via selecting a drop down IP pool). It ignores the other static subnets fine as it should, but that string is empty for some reason.

Does someone have any idea if that is by design or it's a genuine abnormality?

Thanks guys!

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@DanielIonita-3867, Research and find a similar case in internal, the cause of that case is the hardware based firewall between the SCVMM server and the Active Directory Domain controllers did not have Port 464 (TCP\UDP) opened. This port is critical for managing Kerberos passwords. in this case, the firewall between the Failover Cluster nodes and the Active Directory Domain Controllers had Port 464 (TCP/UDP) opened, but it was blocked in the firewall between the SCVMM server and the Domain Controllers. Please check in our environment to see if the port is allowed. Also, make sure all the other ports in the following link are not blocked.
https://docs.microsoft.com/en-us/system-center/vmm/plan-ports-protocols?view=sc-vmm-2019

Please check the ports and if there's any update, feel free to let us know.

0 Votes 0 ·