question

DaveBaddorf-0414 avatar image
0 Votes"
DaveBaddorf-0414 asked DaveBaddorf-0414 answered

How to disable CBT (Change Block Tracking) on a specific Hyper-V VM

Hello!

We started backing up a SQL server with Veeam backup and have had performance problems with the SQL server since that point. The performance/latency issue isn't while the backup is running - it's all of the time.

Since Veeam uses the CBT built into Windows Server 2019 Hyper-V, I was thinking that CBT might be the cause of the poor performance.

I couldn't find anything on how to "officially" turn off CBT for a specific VM, so after experimenting, I found that you could shutdown a VM, remove the .rct and .mrt files and they wouldn't be re-created once the VM was turned back on. (At least until the next backup which would re-enable CBT).

The problem is that after I deleted the .rct and .mrt files from the SQL server, I didn't see any performance improvement. This could be because the problem lies somewhere else (not related to CBT) or it could be that I didn't disable CBT correctly and there is still a performance hit because CBT isn't fully disabled.

Can anyone shed any light on this?

Thanks!

windows-server-2019windows-server-hyper-vwindows-server-clustering
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

DaveBaddorf-0414 avatar image
0 Votes"
DaveBaddorf-0414 answered DaveBaddorf-0414 commented

It turns out that the CBT/RCT wasn't affecting the performance of my VM. I'm sure that there is some performance impact, but it wasn't the cause of my specific issue.

At least reading the posts here, there is no "official" way of disabling CBT/RCT for a specific VM. But it seems to work to shutdown the VM and delete the .rct and .mrt manually.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Another note (and hopefully the last):

Apparently RCT has a memory component on the Hyper-V Server. In other words, from my experimentation, you need to do the following to fully disable RCT on Hyper-V:

  • Shutdown VM

  • Remove .rct and .mrt files.

  • Move VM to another host (then you can move it back if you'd like to)

  • Restart VM

It seems like unless you move the VM to another host (because the host RAM mapping of the changed blocks doesn't transfer), the RCT will still be enabled and potentially affect latency within the VM.

Dave



0 Votes 0 ·
XiaoweiHe-MSFT avatar image
0 Votes"
XiaoweiHe-MSFT answered

Hi,

Thanks for your post, based on my understanding, the issue is that Hyper V VM has performance issue when use Veeam backup. Since performance issue need to analyze performance logs, we are limited to collect the performance logs via forum and the issue is also related with third-party application, which will be complex to troubleshoot, I would suggest you open a case with MS for better help.

Below is the link to open a case with MS:

https://support.microsoft.com/en-us/gp/customer-service-phone-numbers

Thanks for your time!
Best Regards,
Anne


If the Answer is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our [documentation][20] to enable e-mail notifications if you want to receive the related email notification for this thread.
[20]: https://docs.microsoft.com/en-us/answers/articles/67444/email-notifications.html

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

DaveBaddorf-0414 avatar image
0 Votes"
DaveBaddorf-0414 answered XiaoweiHe-MSFT commented

Actually, I wasn't looking for help in troubleshooting the performance issues. I would like like someone to comment on how to remove CBT (Change Block Tracking) for a specific VM.

Does anyone know how to do this? Other than just to remove the .rtc and .mrt files?

Thanks!

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi,

It seems CBT is the mechanism that Veeam used for incremental backup, I found the below article about disbale CBT, please check if it helps:

https://helpcenter.veeam.com/docs/backup/hyperv/changed_block_tracking.html?ver=110

(Please note: Information posted in the given link is hosted by a third party. Microsoft does not guarantee the accuracy and effectiveness of information.)

Best Regards,
Anne



0 Votes 0 ·
DanRhodes-2693 avatar image
0 Votes"
DanRhodes-2693 answered DanRhodes-2693 published

I’m having a very similar problem with a file server. Are you seeing high disk active time in resource monitor? I’m going to look how to turn off CBT if I find a solution I will let you know.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

PascalSlijkerman avatar image
1 Vote"
PascalSlijkerman answered

Hi Dave,

CBT or in Hyper-V known as RTC (Resilient Change Tracker) is a mecanisme to track all changes on block level. This way Veeam knows exactly what to backup instead of scanning/looking for the changes which takes way more time. So during backup you should see a performance improvement.

Seems like you can disable CBT in Veeam. CBT is enabled in the Veeam filter driver on the host. So disabling that might not disable RCT or because RTC is on the host it's already bypasses CBT....But I am no Veeam Expert... For what I know there is no way to disable RTC on Hyper-V. It's enabled by default if you meet the criteria.
https://helpcenter.veeam.com/docs/backup/hyperv/changed_block_tracking.html?ver=110

I think however that your probleem lies somewere else. Do other VMs on the host expierence performance issue's? Are the performance issue's on cpu/mem/disk level in the VM or maybe on the host itself?

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

DaveBaddorf-0414 avatar image
0 Votes"
DaveBaddorf-0414 answered

As a followup, according to this Veeam forum: https://forums.veeam.com/microsoft-hyper-v-f25/windows-server-2019-hyper-v-vm-i-o-performance-problem-t62112-210.html, Microsoft has a long-standing bug with RCT and it's impact on disk performance.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.