How to disable CBT (Change Block Tracking) on a specific Hyper-V VM

Question

Hello!

We started backing up a SQL server with Veeam backup and have had performance problems with the SQL server since that point. The performance/latency issue isn't while the backup is running - it's all of the time.

Since Veeam uses the CBT built into Windows Server 2019 Hyper-V, I was thinking that CBT might be the cause of the poor performance.

I couldn't find anything on how to "officially" turn off CBT for a specific VM, so after experimenting, I found that you could shutdown a VM, remove the *.rct and *.mrt files and they wouldn't be re-created once the VM was turned back on. (At least until the next backup which would re-enable CBT).

The problem is that after I deleted the *.rct and *.mrt files from the SQL server, I didn't see any performance improvement. This could be because the problem lies somewhere else (not related to CBT) or it could be that I didn't disable CBT correctly and there is still a performance hit because CBT isn't fully disabled.

Can anyone shed any light on this?

Thanks!

Answer 1

It turns out that the CBT/RCT wasn't affecting the performance of my VM. I'm sure that there is some performance impact, but it wasn't the cause of my specific issue.

At least reading the posts here, there is no "official" way of disabling CBT/RCT for a specific VM. But it seems to work to shutdown the VM and delete the .rct and .mrt manually.

Answer 2

As a followup, according to this Veeam forum: https://forums.veeam.com/microsoft-hyper-v-f25/windows-server-2019-hyper-v-vm-i-o-performance-problem-t62112-210.html, Microsoft has a long-standing bug with RCT and it's impact on disk performance.

Answer 3

We have been using veeam backup for some while now - no issues was detected during this period. even using CBT in HyperV to backup our ordinary VM's.

We also have a large (20Tb) Virtual HyperV machine - and we only took backup of C: drive (with CBT) - and until now we have, had no performance issues with this - c: drive is only a 100G drive..

However a few days a ago we started to take backup of the large data drives as well - the same minute the backup was initiated the complete server started to have performance issues, and the issues persisted afte the backup completed.

Before we initiated this backup of the large drive, performance was as said fine, but now it sucks.
I have ongoing WMI performance monitoring of all my VMs every 5 min - so I can see/document the behavior and go back the last 30 days to inspec he WMI counters..

or physical disk que size, read transfere, write transferes og both my physical server runnig the VM and the VM it self.

it is clear that the disk que used to be aprox 5000/sec, now they are at 150.000/sec
leading to extremly slow disk IO.
I can see the physical server is not transfering same amount of data, as before - mainly becouse the VM can not fetch the data due to a disk que.

if I move other VM's to the same physical server, these VM's can make normal disk IO -basically telleing me the issues is not the physical server. reather the VHDX files belonging to the bad behaiving VM

all in all, I can confirm that enabling CBT is a bad idea on large/heavy disk IO VM's running under HyperV.

My plan is now to move this vm to vmware, and start using CBT there - hopefully this may solve this issue that MS has had for years and not solved.

Im in the happy situation that I can migrate to vmware cluster, and HyperV/failovercluster will be shurtdown permanently in our production enviroment within a few months :) - I have tested this beta software (HyperV) all to may times, and it keeps having issues not seen in ESXi spending to much time and downtime for by applications.