Hyper-V Changed Block Tracking Troubleshooting

Challenge

Solution

Ensure Veeam Backup & Replication is running on the latest build.  Latest patches will be visible on the download page for Backup & Replication (https://www.veeam.com/vmware-esx-backup/download.html). You will need to login to the Veeam Customer Portal to obtain any patch.
 
For a warning message stating “failed to flush changed block tracking” when backing up VMs on a Hyper-V cluster:

• Ensure none of the cluster nodes are in maintenance mode, have the cluster service stopped, are powered off, or otherwise not accessible.
• Ensure all nodes within the cluster can communicate with each other.  This can be tested by pinging between the nodes and using wbemtest (see http://www.veeam.com/kb1926) to connect between nodes via WMI.  For wbemtest, the two namespaces you want to ensure can be connected to are root\cimv2 and root\virtualization (root\virtualization\v2 for Windows Server 2012+).
• Make sure the hosts are not added to the Veeam console as standalone hosts if they belong to a cluster.  This prevents changed block tracking data from being replicated to all nodes.  The cluster must be added to the Veeam console.

• Ensure the SMB server is added to the Backup & Replication console.

• Ensure the shares are visible in the Backup & Replication console.

• If Hyper-V checkpoints/snapshots are manually created/deleted (merged) on VMs, you will get a message that changed block tracking is not working.  This is because checkpoint/snapshot operations create an additional file (avhd/avhdx) that has to be tracked.  It is added to the list of files to track (or removed in case of deletion/merge of a snapshot/checkpoint) and the changed block tracking warning should resolve the second job run after the snapshot/checkpoint was created/deleted (merged).

• Select the option to disable changed block tracking on all Hyper-V hosts in the cluster.  To do this, right-click on each host in the cluster, select Manage Volumes and then check the box to disable changed block tracking.

 

• Run sc stop VeeamFCT, in an elevated command prompt, on each Hyper-V host in the cluster to ensure the CBT service is stopped.
• Find, export and then delete the following registry key on all Hyper-V hosts in the cluster:  HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VeeamFCT\Parameters\CTDB
• Go back to the Manage Volumes for each host and uncheck the disable changed block tracking option.  Try the job after this.  Allow a couple of job runs before ensuring CBT is working.
• If the issue persists, reboot each Hyper-V host in the cluster.  This will typically involve migrating VMs from one host to another and rebooting the host once it does not own any VMs.  Repeat these steps for all Hyper-V hosts in the cluster.