VMs stored on vVols may experience a significant stun during the snapshot removal process after the backup session.

KB ID: 3055
Product: Veeam Backup & Replication | 9.5 | 10 | 11
Published: 2019-11-22
Last Modified: 2022-07-18
Get weekly article updates
By subscribing, you are agreeing to have your personal information managed in accordance with the terms of Veeam's Privacy Notice.

Cheers for trusting us with the spot in your mailbox!

Now you’re less likely to miss what’s been brewing in our knowledge base with this weekly digest

error icon

Oops! Something went wrong.

Please try again later.

Challenge

During the snapshot removal operation that occurs at the end of a backup/replication job for a VM stored on a vVol Datastore, the following is observed:

  • The Guest OS of the VM being processed is stunned or frozen.
  • Monitoring software may report that the Guest OS is inaccessible due to a lack of ping response.

Cause

The snapshot removal process hangs as it waits for the volume to unbind on the ESXi host. This issue can occur with any ESXi configuration using vVols on vCenter 6.0 and later versions.

This occurs when the following factors are combined:

  • The VM being protected is on a vVol Datastore
  • The Backup Proxy is a VM on a different host than the VM being processed.
  • The Backup Proxy is using the Virtual Appliance (HOTADD) transport mode.

Solution

Below are two solutions for this issue. The first aligns with the recommended solution from VMware, which will require the creation of a Veeam Backup Proxy on every host in the cluster. The second solution does not require the creation of additional proxies and instead introduces a delay between hotadd disk release and snapshot removal request.

VMware Recommended Solution

This solution is based on VMware KB51538, which states:

To resolve this issue place the virtual machine which exists in the vVOL datastore and the virtual machine which involves in the backup process on the same ESXi host. This will ensure that the bind operation is held by the same ESXi host and snapshot consolidation completes without causing long pause.

 

  1. Create a Backup Proxy on every host in the VMware cluster where backups occur.
  2. Create the following registry value on the Veeam Backup Server.

    Key Location: 
    HKLM\Software\Veeam\Veeam Backup and Replication\
    Value Name: EnableSameHostHotaddMode
    Value Type: DWORD (32-Bit) Value
    Value Data:


    For the value data, there are two options, 1 or 2. Both values 1 or 2 will enable a feature that forces Veeam Backup & Replication to first attempt to use, and wait for, the Proxy that is on the same host as the VM to be backed up. The difference between the two is as follows:
    • 1 - If proxy on the same host as VM becomes unavailable, Veeam Backup & Replication will failover to any available proxy and use the available transport mode. This may lead to a situation where a proxy on another host is selected, and hotadd is used, which may cause stun. This ensures the highest performance but may risk VM stun.
    • 2 - If proxy on the same host as VM becomes unavailable, Veeam Backup & Replication will use any available proxy but use network transport mode. This minimizes all stun risk but may reduce backup performance when forced to use Network transport mode.
    Veeam automatically scans for registry values every 15 minutes. Wait 15 minutes for the value to take effect, or stop all jobs and reboot to force the value to be checked.

Alternate Solution

Starting with Veeam Backup & Replication v9.5 U3 (9.5.0.1536), a setting was added to allow for the configuration of a delay between the detachment of hotadded disks and the request for snapshot removal. This delay can be configured using the following registry value:

Create On: Veeam Backup Server
Key Location: HKLM\SOFTWARE\Veeam\Veeam Backup and Replication\
Value Name: HotaddTimeoutAfterDetachSec 
Value Type: DWORD (32-bit) Value 
Value Data: 60

The delay is measured in seconds and may need additional adjustment based on the environment. In most cases, 60 seconds has been plenty of time for disks to detach from the proxy before the snapshot removal request was issued. A smaller delay may also work. However, a larger delay may be required in some environments.

Click here to send feedback regarding this KB, or suggest content for a new KB.
To report a typo on this page, highlight the typo with your mouse and press CTRL + Enter.

Spelling error in text

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Thank you!

Thank you!

Your feedback has been received and will be reviewed.

Oops! Something went wrong.

Please try again later.

KB Feedback/Suggestion

This form is only for KB Feedback/Suggestions, if you need help with the software open a support case

By submitting, you are agreeing to have your personal information managed in accordance with the terms of Veeam's Privacy Notice.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Verify your email to continue your product download
We've sent a verification code to:
  • Incorrect verification code. Please try again.
An email with a verification code was just sent to
Didn't receive the code? Click to resend in sec
Didn't receive the code? Click to resend
Thank you!

Thank you!

Your feedback has been received and will be reviewed.

error icon

Oops! Something went wrong.

Please try again later.