Challenge
VMware backup jobs fail after update 3, whereas prior to update they have been working.
- The jobs are using Network transport mode
- Source Proxy is Windows 2008SP1/Windows 2008R2
The error message for all jobs in the GUI looks like this one:
Processing VM_Name Error: Failed to open VDDK disk [datastore] VM_Name/VM_Name.vmdk] ( is read-only mode - [true] ) Logon attempt with parameters [VC/ESX: [hosname.local];Port: 443;Login: [administrator@vsphere.local];VMX Spec: [moref=vm-ref];Snapshot mor: [snapshot-ref];Transports: [nbd];Read Only: [true] failed because of the following errors: Failed to open disk for read. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.
Solution
This issue is related to some of the dll files in the Windows ZIP file for VDDK 6.5 with incorrect manifests, making them depend on two Visual C++ 9.0 versions, both 9.0.32709.4148 and 9.0.21022.8. This causes a failure to load vixDiskLibVim.dll with error code = 0x36b1, which means side-by-side configuration is incorrect.
Workaround
Edit failing jobs and configure them to use a proxy running Windows 2012/2012R2/2016 proxy.
Note: The steps to Verify and Correct this issue below are to be performed on the proxy which was used by the failing job.
Verify issue matches this KB:
- On the source proxy navigate to Veeam logs location (default %ProgramData%\Veeam\Backup)
- Open the folder that matches the name of the failing Job.
- Find the log file matching the nomenclature Agent.Job_Name.Source.VM_Name.vmdk.log.
- Open the file with any text editor and search for:
VDDK error: 16053 (One or more required subsystems failed to initialize)
Example snippet:[DD.MM.YYYY HH:MM:SS] < 12620> cli| ERR |Command 'Open' has failed. [DD.MM.YYYY HH:MM:SS] < 12620> cli| >>|VDDK error: 16053 (One or more required subsystems failed to initialize). Value: 0x0000000000003eb5 [DD.MM.YYYY HH:MM:SS] < 12620> cli| >>|--tr:Failed to open virtual disk [Test_Vol1_LUN3] TEST/TEST.vmdk (flags: 4) [DD.MM.YYYY HH:MM:SS] < 12620> cli| >>|An exception was thrown from thread [12620].
- To further confirm, open event viewer and in Application logs search, Source: 'SideBySide' Event ID: '35'
example event:
How to correct this issue:
- Open Programs and Features and verify that Microsoft Visual C++ 2008 Redistributable Package (x64) is not installed.
- Download and install Microsoft Visual C++ 2008 Redistributable Package (x64) (https://www.microsoft.com/en-us/download/details.aspx?displaylang=en&id=15336)
- Ensure that no other jobs are running, and reboot the proxy.
- Retry failed jobs.
More Information
Note: This KB article only applies if VDDK error 16053 is found in the logs. If this error is not found in the logs, please contact Veeam support for further troubleshooting.