- Veeam Support Knowledge Base
- VMware Instant Recovery Times Out
VMware Instant Recovery Times Out
Cheers for trusting us with the spot in your mailbox!
Now you’re less likely to miss what’s been brewing in our knowledge base with this weekly digest
Challenge
Instant VM recovery of a VMware VM fails after at least 30 minutes with the error:
Failed to publish VM
Cause
Veeam Backup & Replication implements timeouts for most operations to protect against hangs. However, even when no process is hung, timeouts may occur due to significant performance problems or an unusual use case.
Typically, this error occurs due to the slow performance of the vPower NFS datastore. Possible causes of the slow performance:
- Slow repository read performance, mainly due to deduplication storage "rehydrating" deduped/compressed backup data.
- Slow network link between host and vPower NFS server due to congestion, setting of 100Mb/s on NIC, or other infrastructure issues.
- Poor performance of the vPower NFS server.
Solution
Increase Timeouts
Increase the timeouts below to several times the default value. Although you can increase the timeouts beyond these limits, it is usually better to investigate performance first if you are performing a test or low-priority restore. In an emergency, increase the timeouts as high as necessary, or create a Severity 1 case with Veeam Support for assistance in finding a workaround.
All values are created on the Veeam Backup Server. Some may not exist already and must be created.
All values are in decimal. Make sure no jobs, restores, or other tasks are running, then restart the Veeam Backup Service to apply these changes.
RemotingTimeout
Controls general timeout between all services.
Key Location: HKLM\SOFTWARE\Veeam\Veeam Backup and Replication
Value Name: RemotingTimeout
Value Type: DWORD (32-Bit) Value
Value Data Default: 900 (seconds)
Value Data Suggested: 7200
New-ItemProperty -Path 'HKLM:\SOFTWARE\Veeam\Veeam Backup and Replication\' -Name 'RemotingTimeout' -Value 7200 -PropertyType DWORD -ForceIrMountLeaseTimeOut
This applies to mounting VMware virtual disks from backup or storage snapshots for instant VM recovery, except when restoring to vCloud Director.
Key Location: HKLM\SOFTWARE\Veeam\Veeam Backup and Replication
Value Name: IrMountLeaseTimeOut
Value Type: DWORD (32-Bit) Value
Value Data Default: 30 (minutes)
Value Data Suggested: 90
New-ItemProperty -Path 'HKLM:\SOFTWARE\Veeam\Veeam Backup and Replication\' -Name 'IrMountLeaseTimeOut' -Value 90 -PropertyType DWORD -ForceIrVcdMountLeaseTimeOut
This applies to mounting VMware virtual disks from backup for instant VM recovery to vCloud Director.
Key Location: HKLM\SOFTWARE\Veeam\Veeam Backup and Replication
Value Name: IrVcdMountLeaseTimeOut
Value Type: DWORD (32-Bit) Value
Value Data Default: 45 (minutes)
Value Data Suggested: 120
New-ItemProperty -Path 'HKLM:\SOFTWARE\Veeam\Veeam Backup and Replication\' -Name 'IrVcdMountLeaseTimeOut' -Value 120 -PropertyType DWORD -ForceNote: Additional considerations and timeout values may be applicable to restore from storage snapshots.
Improve Performance
The most common cause of these errors is slow read performance from the backup repository. Deduplicating storage is not recommended as a back-end for vPower-based restores. Where possible, optimize storage devices for random read I/O of large blocks (typically 256 KB – 512KB with default settings or 4 MB for backups on deduplicating storage; your use case may vary). A simple benchmark is described in KB2014.
As a workaround or to verify that storage performance is the cause of the timeout, try temporarily moving the backup files to faster storage.
Where possible, ensure the vPower NFS server, the repository, and the destination ESXi host for the restore are all located at the same site. Avoid creating unnecessary bottlenecks by sending restore traffic over the WAN.
Additional Performance Troubleshooting
- Depending on the underlying infrastructure, significant performance differences between running the vPower NFS service from a VM or a physical machine can exist. For example, try using a VM located on the same ESXi host as the virtual lab.
- Heavily fragmented full backup files can reduce restore performance. Schedule compact operations to minimize fragmentation.
- Where applicable, test the throughput of the network connections between the repository and the vPower NFS server and between the vPower NFS server and the ESXi host.
- Investigate CPU and memory usage of the repository and vPower servers.
More Information
User Guide:
The remotingTimeout setting affects all processes and services communicating with the Veeam Backup Service. Communication failures will sometimes be retried, so an operation may not fail until this timeout has occurred multiple times.
Consider that, from a networking and vSphere configuration perspective, there is little difference between vPower and any other NFS datastore.
VMware Technical Paper: Best Practices for Running vSphere on NFS Storage
To report a typo on this page, highlight the typo with your mouse and press CTRL + Enter.
Spelling error in text
Thank you!
Your feedback has been received and will be reviewed.
Oops! Something went wrong.
Please try again later.
You have selected too large block!
Please try select less.
KB Feedback/Suggestion
This form is only for KB Feedback/Suggestions, if you need help with the software open a support case
Thank you!
Your feedback has been received and will be reviewed.