System Center Virtual Machine Manager(SCVMM) is using BITS instead of Fast File Copy while creating a new VM
So far you would have seen Blog Posts from me only on Microsoft Security Products like ISA, TMG, UAG and FIM. I have recently ventured into the wonderful world of SCVMM.
It’s an Awesome Product to work on. Recently I worked on an interesting case, so thought of sharing it with you all. Just to give you a brief background, we were trying to Create a new VM and we were seeing that VMM was using BITS for copying the VHD whereas it should be using Fast File Copy instead as per the new Advancement introduced in SCVMM 2012 R2.
Here are all the Details of the issue and how did we nailed it down and what was the Fix:
When using SCVMM to deploy virtual machines ODX (Fast File Copy) was not being used to copy the VHDs from the library to the NetApp SMB 3.0 share. SCVMM would copy the data over using BITS over HTTPS resulting in average virtual machine deployment time of 17 minutes.
This is an intra-svm / intra-volume copy operation.
We could verify that the NetApp was set up properly to handle ODX by invoking the same copy manually (not using SCVMM) - below is a screen shot showing a data copy invoked manually for the same VHDs from the SCVMM library going to the same NetApp SMB 3.0 share where VMs will reside. This screen shot shows that the data transfer is moving at 149MB/s and not registering any traffic on the underlying network interface - this is an acceptable speed given it is an intra-svm transfer.
Steps Taken to Resolve Issue:
We first engaged NetApp support wondering if there was a misconfiguration on the NetApp topology keeping ODX from working. After a couple phone calls and config reviews we found that everything on the NetApp was set up properly and should be working.
We verified that the storage / Hyper-V and SCVMM topology should support Offloaded Data Transfers for the VMM Library during virtual machine deployment. We then moved to capturing VMM diagnostics logs while we re-produced the issue.
NOTE: To know how to collect VMM Tracing and how to Convert those ETL Traces to Readable Text File, please refer the Article below:
https://support.microsoft.com/en-us/kb/2913445
I looked at the VMM tracing and here is what I saw:
7887 6768,03:29:06.171 03-24-2015,0x0EC4,0x0FDC,16,NewVmSubtaskBase.cs,3143,0x00000000,Using network deployment type WindowsCopy,{00000000-0000-0000-0000-000000000000},4,
8054 6935,03:29:06.538 03-24-2015,0x0EC4,0x0FDC,4,WindowsCopyCredentialFixup.cs,278,0x00000000,CarmineException [ex#14a0] caught by SetFastFileCopyCredentials (catch CarmineException) [[(CarmineException#1783c) { Microsoft.VirtualManager.Utils.CarmineException: This operation requires the library server LibraryServer.Contoso.com to have a management credential to access file share.
8056 at Microsoft.VirtualManager.Engine.VmOperations.WindowsCopyCredentialFixup.GetLibraryCredentialsForItem(Guid libraryServerId; String itemLocation; String destination)
8057 at Microsoft.VirtualManager.Engine.VmOperations.WindowsCopyCredentialFixup.SetDeploymentFileLibraryCredentials(DeploymentFile file; String destination)
8058 at Microsoft.VirtualManager.Engine.VmOperations.WindowsCopyCredentialFixup.SetFastFileCopyCredentials(List`1 deploymentFiles)
8065 at Microsoft.VirtualManager.Engine.VmOperations.WindowsCopyCredentialFixup.GetLibraryCredentialsForItem(Guid libraryServerId; String itemLocation; String destination)
8066 at Microsoft.VirtualManager.Engine.VmOperations.WindowsCopyCredentialFixup.SetDeploymentFileLibraryCredentials(DeploymentFile file; String destination)
8067 at Microsoft.VirtualManager.Engine.VmOperations.WindowsCopyCredentialFixup.SetFastFileCopyCredentials(List`1 deploymentFiles)
8083 6936,03:29:06.538 03-24-2015,0x0EC4,0x0FDC,16,WindowsCopyCredentialFixup.cs,279,0x00000000,Could not set FFC credentials for file \\ LibraryServer.Contoso.com\MSSCVMMLibrary\VHDs\WAP-WS2012R2-DC-Gen1.vhdx,{00000000-0000-0000-0000-000000000000},1,
8110 6963,03:29:06.563 03-24-2015,0x0EC4,0x0FDC,16,WindowsCopyCredentialFixup.cs,295,0x00000000,Reverting credentials for file \\ LibraryServer.Contoso.com\MSSCVMMLibrary\VHDs\WAP-WS2012R2-DC-Gen1.vhdx,{00000000-0000-0000-0000-000000000000},2 ,
8111 6964,03:29:06.563 03-24-2015,0x0EC4,0x0FDC,16,NewVmSubtaskBase.cs,3438,0x00000000,Deployment type WindowsCopy for VM testodx002 no longer possible; attempting fallback to BITS deployment for VM files,{00000000-0000-0000-0000-000000000000},1,
8202 at Microsoft.VirtualManager.Engine.VmOperations.NewVmSubtaskBase.DeployFilesWithFallback(DeploySubtask mainSubtask; DeploymentHelper mainHelper; DeploySubtask fallbackSubtask; DeploymentHelper fallbackHelper; WindowsCopyCredentialFixup credentialFixup; Boolean useFallback; Boolean isSubtaskAlreadyChild; String traceMessage)
8252 at Microsoft.VirtualManager.Engine.VmOperations.NewVmSubtaskBase.DeployFilesWithFallback(DeploySubtask mainSubtask; DeploymentHelper mainHelper; DeploySubtask fallbackSubtask; DeploymentHelper fallbackHelper; WindowsCopyCredentialFixup credentialFixup; Boolean useFallback; Boolean isSubtaskAlreadyChild; String traceMessage)
So as we can see in the VMM trace above that this operation requires Library Server to have Management Credentials for the File Share.
When we checked the settings of the Library Server on SCVMM Console, it had no management credential assigned as seen below:
So we assigned the appropriate user account in the Library Management Credentials Settings and Once we implemented this change the VMM VM deployment would not attempt to use Offloaded Data Transfer but would fail to use ODX and fall back to BITS over HTTPS:
We performed another diagnostics capture on SCVMM while reproducing this issue and this time the analysis showed this:
11031 [0]0EC4.1544::2015-03-24 12:31:49.576 [Microsoft-VirtualMachineManager-Debug]4,1,WindowsCopyDeployer.cs,293,GetFinalResult() for deployment of file \\LibraryServer.Contoso.com\MSSCVMMLibrary\VHDs\WAP-WS2012R2-DC-Gen1.vhdx to \\TENNTAP1CSVM01\TEN_SMBShare01\testodx012\WAP-WS2012R2-DC-Gen1.vhdx returned 2147942405,{00000000-0000-0000-0000-000000000000}
11050 [1]0EC4.1544::2015-03-24 12:31:49.589 [Microsoft-VirtualMachineManager-Debug]4,2,BitsDeployer.cs,974,CarmineException [ex#1895] caught by GetProgress (catch CarmineException) [[(CarmineException#19068) { Microsoft.VirtualManager.Utils.CarmineException: VMM could not transfer the file
\\LibraryServer.Contoso.com\MSSCVMMLibrary\VHDs\WAP-WS2012R2-DC-Gen1.vhdx to \\TENNTAP1CSVM01\TEN_SMBShare01\testodx012\WAP-WS2012R2-DC-Gen1.vhdx using fast file copy. The VMM agent on host TENHV1N01.contoso.com returned an error. Ensure that source file is accessible to VMM and that the destination is a valid path.
Microsoft.VirtualManager.Engine.Deployment.WindowsCopyClient.GetProgress(UInt32& state, UInt64& filesizeBytes, UInt64& transferredBytes) at Microsoft.VirtualManager.Engine.Deployment.BitDeployer.GetProgress(InProgressDeploymentJob runningJob, UInt32& state, UInt64& fileSizeBytes, UInt64& transferredBytes) at Microsoft.VirtualManager.Engine.Deployment.BitDeployer.GetProgress(UInt32& clientState) *** Carmine error was: FastFileCopyCouldNotCopyFile (26559); HR: 0x80070005 *** \\LibraryServer.Contoso.com\MSSCVMMLibrary\VHDs\WAP-WS2012R2-DC-Gen1.vhdx ** \\TENNTAP1CSVM01\TEN_SMBShare01\testodx012\WAP-WS2012R2-DC-Gen1.vhdx ** TENHV1N01.contoso.com ** [s#5de060] Task`1.SubtaskRun(this: (Task`1#a), TaskID: (guid) f2b5dd60-5baf-44d7-ba9b-5bb524a9e735) in Task.cs:line 252 Full call stack from when the exception was thrown: at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo) at System.Environment.get_StackTrace() at
11057 [1]0EC4.1544::2015-03-24 12:31:49.594 [Microsoft-VirtualMachineManager-Debug]16,1,NewVmSubtaskBase.cs,3478,Could not deploy files for VM testodx012 using WindowsCopy, attempting fallback to BITS deployment for VM files,{00000000-0000-0000-0000-000000000000}
11072 [0]0EC4.1544::2015-03-24 12:31:49.602 [Microsoft-VirtualMachineManager-Debug]16,2,WindowsCopyCredentialFixup.cs,295,Reverting credentials for file \\LibraryServer.Contoso.com\MSSCVMMLibrary\VHDs\WAP-WS2012R2-DC-Gen1.vhdx,{00000000-0000-0000-0000-000000000000}
The key we found here was that the SCVMM management credential did not have sufficient access to the SCVMM library share. We gave the account Full Permission to the SCVMM Library Share.
After implementing this change ODX worked as advertised on both SCVMM and the NetApp - The Deploy file operation in the job would use ODX (Fast File Copy):
Conclusion
Sometimes an issue can appear to be something highly complex when you think about each layer of the environment and the potential for misconfigurations:
· Intra-SVM Offloaded Data Transfer on NetApp
· Fibre Channel Virtualization
· SMB 3.0 on NetApp
· The fact that we are using ODX across two storage protocols (SMB and vFC)
· SCVMM Cluster
· File Server Cluster accessing shared storage via Virtual Fibre Channel
In this particular case it was something as simple as a management credential (Run-As account) being defined to manage the SCVMM Library Share and the appropriate permissions for that management credential to access the share.
I Hope that this Post will be helpful for all the SCVMM Admins. Will keep coming with such similar Posts in the future as well.
I would like to Specially Thank Nick Hawkins Twitter:@nhawkins with whom I worked on this case, for providing us with all the above details on this issue. Only because of that I was able to share it with you.
Nick has his own Blog Site as well https://www.nickahawkins.com . He has written some amazing Posts on different Technologies/Products. Please have a look at his Blog Site as well and I am sure it will prove to be very useful for you.
Author:
Nitin Singh
Support Escalation Engineer
Microsoft Security and Manageability Division