Failed to generate VHD tree

Question

We are running a HyperVisor on Microsoft Windows Server 2016 Standard 10.0.143939 Build 14393 and on it 4 virtual machines also running the same Windows version.

All 4 VM's are included in the back-up with Veeam Backup & Replication 9.5 Update 4 (9.5.4.2866).
Right now 3 of the 4 are going well with Veeam. However one is creating problems.

At one point I as busy with updates on the Hypervisor while only 1 VM was running while doing it. Unfortunately at one time it needed several reboots and one time the shutdown of the VM was not done.
After a reboot I noticed an avhdx file and extra set of mrt en rct files.
The next day all was still wel and I decided to merge the ahvdx file with its parent vhdx file to get it back to normale state.
No problem there! After succesfully merging it the avhdx file was removed, however the mrt and rct file were still there.

The day after Veeam tried to backup this VM and it starting to create problems. It generated the following events on the Hypervisor:
Event ID: 20864 Source: Hyper-V-VMMS: Virtual machine failed to generate VHD tree: 'Catastrophic failure'('0x8000FFFF').
Event ID: 14320 Source: Hyper-V-VMMS: Cannot delete checkpoint: Catastropic failure (0x8000FFFF). Checkpoint ID <identifier>.
Event ID: 15070 Source: Hyper-V-VMMS: 'Servername' failed to remove checkpoint. (Virtual Machine ID <identifier)
Event ID: 16010 Source: Hyper-V-VMMS: The operation failed.
Event ID: 20864 Source: Hyper-V-VMMS: Virtual machine failed to generate VHD tree: 'Catastrophic failure'('0x8000FFFF').

This generated again an ahvdx, ahvdx.mrt and avhdx.rct for every time it tried to back-up. I checked with Powershell that all the Snapshottypes are off Recovery and 1 Standard.

Veeam says it is an Microsoft/Hyper-V issue and they can't fix it.

I would like to know what else I need to do besides merging every recovery checkpoint. How do I get rid of all those extra files.
Note: This is a domain server with AD, DNS and DHCP so I don't want to start over with a new VM and restore an old back-up.
Can anyone give a detailed step by step guide what to do.
The goal is only the Standard Snapshot and a woring Back-up again.

Thanks in advance.

Answer 1

Hi,
According to your description, the avhdx file and extra set of mrt rct files are possibly created because backup and replication with RCT.
(https://helpcenter.veeam.com/docs/backup/hyperv/changed_block_tracking.html?ver=100)
And you can try the following steps to check if it works:
https://blog.workinghardinit.work/2015/10/15/remove-lingering-backup-checkpoints-from-a-hyper-v-virtual-machine/
(Please note: Information posted in the given link is hosted by a third party. Microsoft does not guarantee the accuracy and effectiveness of information.)

I also find some similar problems which may be helpful for you:
https://social.technet.microsoft.com/Forums/en-US/89f62a9f-be28-4248-80c3-21d8a4d1e1d8/avhdx-leftover-from-backup-tool-how-to-merge-or-delete-those-files?forum=winserverhyperv
https://community.spiceworks.com/topic/632113-hyper-v-2012-r2-checkpoint-backups-how-to-delete?page=1
https://social.technet.microsoft.com/Forums/windows/en-US/10485462-ccdf-4ee8-8c28-5748e46ef6ce/virtual-machine-failed-to-generate-vhd-tree-catastrophic-failure0x8000ffff?forum=w8itprovirt

Thanks for your time!
Best Regards,
Mico Mi

-----------------------------

If the Answer is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

Answer 2

Right now I merged all the disks correctly to its parent with the edit disk function.
However now I can still see all the previous checkpoints in the tree. They should be gone!

How do I get rid of them?

Answer 3

Hi,
Did you try to remove the lingerging backup checkpoints via powershell?
Get-VMSnapshot -ComputerName "MyHyperVHost" -VMName "VMWithLingeringBackupCheckpoint" | Remove-VMSnapshot
Thanks for your time!
Best Regards,

Mico Mi

If the Answer is helpful, please click "Accept Answer" and upvote it.

Answer 4

I did now, however your command is not correct:
It should be:
Get-VMSnapshot -ComputerName "HyperVHost" -VMName "VMName" -Name "Snapshot you want to delete" | Remove-VMSnapShot

It come sback with the same error as I press delete on the snapshot:
The operation failed because the file was not found.

Sounds like I have to export or take it offline and create the VM again with the vHDx file.