Failback That Was Canceled or Failed  Results in CID Mismatch

KB ID: 2113
Product: Veeam Backup & Replication
Published: 2016-03-21
Last Modified: 2022-06-18
Get weekly article updates
By subscribing, you are agreeing to have your personal information managed in accordance with the terms of Veeam's Privacy Notice.

Cheers for trusting us with the spot in your mailbox!

Now you’re less likely to miss what’s been brewing in our knowledge base with this weekly digest

error icon

Oops! Something went wrong.

Please try again later.

Challenge

In rare situations, if a VMware Replication Failback to Original VM fails or is canceled, the original VM may be left in a ‘consolidation needed’ state and cannot be powered on or consolidated because of errors such as:
The parent virtual disk has been modified since the child was created
or
Content ID mismatch
When this occurs, if a second Failback targeting the original VM is attempted, it will fail because the disk configuration on the Original VM is invalid. The disks on the Original VM will appear to be 0B in size, which causes the Failback to fail with the error:
Failed to perform failback Error: Destination VM and replica VM disk sizes are different.

Cause

When the Failback fails or is canceled, Veeam Backup & Replication sends requests to the VMware environment to remove the fallback snapshot that was created on the original VM to return it to the pre-failover state. However, an issue within ESXi causes the snapshot to not be removed correctly, leaving the Original VM in a state where it may appear as if it is corrupted and cannot be Powerd On.

Solution

Contact Veeam Support

This article's solution section contains Advanced Troubleshooting techniques that, if performed incorrectly, could damage the original VM.

We strongly advise that if you are unsure about your ability to perform these steps and would like assistance, please create a case with Veeam Support and mention this KB article.

If production data is involved, it is crucial to understand the differences between Failover and Planned Failover and the impact that Failback to Orginal VM will have.

  • Failover, or what can be referred to as "Unplanned" Failover, is intended to be used when the original VM that the Replica was based on is either unavailable or its data is not useable.
  • Planned Failover is utilized to either perform a planned migration of a VM between locations or as part of a failover test. The Planned Failover option will first synchronize the original VM and replica states, then power on the Replica VM.

Performing a Failback to the Original VM will copy the current state of the Replica VM to the original VM. Meaning, that if the Failover option (as opposed to Planned Failover) was used while the original VM was in a stable functioning state, attempting to then Failback from that Failover state would cause the Replica's state to overwrite the Original VM. This will lead to a loss of all changes that have occurred on the original VM between when the last Replication job run occurred and the Failover was performed.

Scenario Specific Information

Cancelled Failback

If the failback was canceled, and there is a need to revert the original VM to the pre-failback state safely, please create a case with Veeam Support so that our Support Staff can assist with this situation.

If there is no need to safely revert the original VM to the pre-failback state, proceed with reviewing the options below.

 

Testing Planned Failover or Intentional Failover due to Corrupted Original VM

If Planned Failover was used to perform a failover test or the Failover was performed because the original VM was unfit for service, there are three options to proceed:

If the Failback procedure failed, you will need to create a case with Veeam Support to investigate why the Failback failed. The procedure documented below will only fix the Original VM. It will not correct the underlying reason Failback failure issue.

Repairing Original VM to Reattempt Failback

There are two options to resolve the ‘consolidation needed’ status and return the original VM to a state where it can be used as a Failback target:

Option 1: Resolve CID Mismatch and Consolidate Partial Data

If you wish to continue trying to fail back to the original VM and keep the partial data already transferred to minimize the amount of data to be transferred during the next failback attempt, follow the steps in VMware KB: 1007969 to resolve the underlying CID mismatch. Alternatively, if keeping the transferred data is not important, or the steps in the VMware article do not work or are too complicated, try the alternative steps listed in Option 2.

Note that if the production VM was stable before the Planned Failover and subsequent Failback, performing the steps in  VMware KB: 1007969 will merge the partial failback data into the base disks of the Original VM, leaving that VM in an incomplete state. To repair the Original VM, you must complete the failback process to synchronize the stable state of the replica to the Original VM.

Option 2: Manually Revert Original VM Disk Configuration

If you want to revert the original VM to its pre-Failback state, you will perform a variation on the "Alternative procedure" from  VMware KB: 1007969, described below. This procedure involves disconnecting the snapshot disks and reattaching the original base disks, somewhat similar to Veeam KB1773, except that instead of modifying the Replica's disks, you'll modify the Original VM. Performing this will cause all data within the snapshots on the Original VM to be lost, including data that was present in a snapshot created before the Planned Failover and Failback.

If you are uncomfortable completing these steps, please stop and create a case with Veeam Support referencing this KB.

 

Gather Information

  1. Edit the original VM using the vSphere client.
  2. Note the SCSI node assignments for each disk file and the location of those disk files.

    Example:
    SCSI0:0 = [Datastore1] DC01\DC01-000001.vmdk
    SCSI0:1 = [Datastore1] DC01\DC01_1-000001.vmdk
    SCSI0:2 = [Datastore2] DC01\DC01-000001.vmdk
     

Detach Snapshot Disks and Attach Base Disks

  1. Edit the Original VM, and detach/remove each of the snapshot disks.
  2. After removing all of the snapshot disks from the Original VM, edit it again and reattach the base disks to each SCSI node.
    1. Select the Add Existing Hard Disk option, then navigate to the location of the base disk.
    2. After the disk is attached, assign the correct SCSI node based on the path and VMDK file name.

      For example, if the snapshot disk [Datastore1] DC01\DC01-000001.vmdk was using SCSI0:0, then the base disk would be 
      SCSI0:0 = [Datastore1] DC01\DC01.vmdk and it should be assigned to that same SCSI node.
    3. Repeat the above 2 steps to attach each of the base disks to the same SCSI nodes noted earlier.

Datastore Cleanup

  1. Using the datastore browser, navigate to the Original VM's folder.
  2. Delete all child disk files created by the snapshot (vmname-000###.vmdk) and any –ctk.vmdk files. Also, delete VMSN and VMSD files.
  3. If there were multiple datastore folders among the disks, check those other datastores as well.

Test the VM

  1. Create a snapshot on the Original VM.
  2. Open the Snapshot Manager and click the Delete All button.
  3. If the Snapshot Removal completes without error, the disks are reverted.

 

If you completed the procedure correctly, the Original VM should now be in the same state it was in before the Failback. You may now proceed with reattempting the Failback procedure if you want to synchronize the Replica's state over the Original VM.

Note: If the previous Failback operation failed, you should create a case with Veeam Support to investigate why that Failback failed. The procedure documented above will only fix the Original VM. It will not correct the underlying reason the Failback failed.

To submit feedback regarding this article, please click this link: Send Article Feedback
To report a typo on this page, highlight the typo with your mouse and press CTRL + Enter.

Spelling error in text

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Thank you!

Thank you!

Your feedback has been received and will be reviewed.

Oops! Something went wrong.

Please try again later.

KB Feedback/Suggestion

This form is only for KB Feedback/Suggestions, if you need help with the software open a support case

By submitting, you are agreeing to have your personal information managed in accordance with the terms of Veeam's Privacy Notice.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Verify your email to continue your product download
We've sent a verification code to:
  • Incorrect verification code. Please try again.
An email with a verification code was just sent to
Didn't receive the code? Click to resend in sec
Didn't receive the code? Click to resend
Thank you!

Thank you!

Your feedback has been received and will be reviewed.

error icon

Oops! Something went wrong.

Please try again later.