VDDK error: 13 - Troubleshooting

Backup/Replication jobs fail with:

VDDK error: 13.You do not have access rights to this file

Below is a list of possible solutions to this issue sorted by what transport mode was being using when this error occurred.
Note: There are many causes for VDDK 13, this list is not intended to be all encompassing.

Using Transport Mode: Network Mode

• Solution 1:
  In some instances, you may receive this error when permissions are not correctly set. To confirm, please follow the steps outlined in the following KB article to test ports, permissions, and DNS resolution
  http://www.veeam.com/kb1198

Using Transport Mode: Hotadd Mode

• Solution 1:
  https://www.veeam.com/kb1054
  VMware Tools must be installed and up-to-date on the Veeam server and any Veeam proxies
• Solution 2:
  https://www.veeam.com/kb1775
  Stuck Hotadded Disks – Review all virtual proxies for disks attached that do not belong to the VM in question, remove the disks not owned by this VM without deleting from the datastore. Once deleted, consolidate snapshots on the affected VMs.
• Solution 3:
  https://kb.vmware.com/s/article/1002310
  Orphaned Snapshots – Consolidate snapshots on affected VMs
• Solution 4:
  http://www.veeam.com/kb1875
  Permissions - Please configure the account leveraged to access the vCenter as an Administrator within vCenter
• Solution 5:
  http://www.veeam.com/kb1882
  Automount – In some cases, it may be necessary to perform an automount scrub to clear any existing mount points that are no longer in the system.
  
  Run the following via CMD Prompt on the proxy server:

  diskpart
  automount disable
  automount scrub
  exit
  

• Solution 6:
  http://www.veeam.com/patches.html
  Patching – Please ensure that all VMware and Veeam components are fully up to date.
• Solution 7:
  In some rare instances, performing a storage vMotion has been known to resolve the issue due to a locked VM file on the datastore where the VM files reside.
• Solution 8:
  In some rare instances, restarting the vCenter appliance or alternatively restarting the ESXi management agents (https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003490) may yield positive results if all above testing does not resolve the issue.
• Solution 9:
  Veeam Proxy has a Veeam replica with the same bios UUID within the vCenter Server. The following can be observed in 
  
  Job.Job_name.log:

  [time stamp] < threadID> Info VM VeeamProxy _replica(vm-100001) has the same bios uuid as proxy VeeamProxy.local in the DB
  

  Agent.Job_name.VM_name.Hotadd.Attacher.log:

  [time stamp] < threadID> vdl| [vddk] time stamp error -[07132] [Originator@6876 sub=transport] Cannot use mode hotadd to access [datastore_name] VM_name/VM_name.vmdk: Cannot mount using this method. (Mounting VM vm-101 using transport hotadd failed : Cannot access datastore for one of the disks of Virtual Machine VM_name..)
  

  The behavior is observed when a Veeam Proxy is replicated with Veeam B&R, the replication process preserves the original VM bios UUID for restore purposes. To work around the issue setup a dedicated VM as a Veeam Proxy, replicating Veeam Proxies is not usually necessary due to ease of deployment. More information: https://www.veeam.com/kb2159

• Solution 10:
  In some rare cases you may encounter a limitation for the amount of disks attached to virtual SCSI controller (max. 15 disks). In this case you may try to add another SCSI controller to proxying server in use by the job. Please keep in mind that there's a limitation for the amount of SCSI controllers - 4 per VM.
  
• Solution 11:
  Known issue with vSphere 6 or later when a proxy has a SATA Controller present.
  (https://kb.vmware.com/s/article/2151091)
• Solution 12:
  
  In some rare instances, while using Hot-add Mode, the disk is successfully added to the proxy VM, however, the disk is not visible under Disk Management. No errors appear in Event Logs regarding the disk being added, and no errors show on vSphere. In this situation, Hot-Add mode cannot be used due to an issue with the virtual SCSI controller.
  
  In order to identify this specific scenario:
  
  1) Perform a manual hot-add: https://www.veeam.com/kb1184
  2) Task will be successful on vSphere side. The disk is visible within the proxy VM settings in vSphere, however, the added disk does not appear in disk management of proxy VM.
  3) The issue is related to a VMware infrastructure issue, review the list VMware article below. If this issue persist, it is advisable to contact to VMware Support for further investigation.
  
  • https://kb.vmware.com/s/article/2135969
  • https://kb.vmware.com/s/article/2048497
  • https://docs.vmware.com/en/VMware-Workstation-Player-for-Windows/16.0/com.vmware.player.win.using.doc/GUID-7F8982C6-EBC1-4338-BA8C-A83952BB2E4C.html

Using Transport Mode: Direct SAN

• Solution 1:
  When receiving VDDK Error 13 when leveraging DirectSAN access mode, please confirm that the LUNs are presented to the proxy correctly and have the proper permissions. For assistance deploying & troubleshooting DirectSAN please follow the below KB articles:
  http://www.veeam.com/kb1895

Additional Troubleshooting(All versions):

• Manual Hotadd - http://www.veeam.com/kb1184
• Hotadd Requirements - http://www.veeam.com/kb1054
• Deploy an additional standalone proxy to isolate failures to a single specified proxy
• To isolate a specific transport mode as the point of failure, you may also test the other transport modes to ensure that connectivity is correctly established and backup jobs work as intended via the other transport mode.

Additional Troubleshooting (vSphere 5.1 and below):
http://www.veeam.com/kb1777
Backup via Standalone host – Configure backup job to backup via Standalone host rather than through vCenter; this will isolate vCenter as the point of failure. If backup via standalone host is successful, you can attempt restarting the vCenter as well as the management agents on the host(s) where the proxies/VM(s) reside.

Information: Hotadd will only work when the proxy is on the same host as the object being backed up

Additional Troubleshooting (vSphere 5.5):
When attempting to hotadd leveraging a standalone host configuration (that is managed by a vCenter object) with vSphere 5.5 you will receive VDDK Error 13, as well as the error listed below:

Error message: [00680 error 'transport'] Reconfigure attempt failed for VM "2" with Vmomi::MethodFault::Exception ("vim.fault.HostAccessRestrictedToManagementServer")\n

To resolve this particular issue, please leverage Network Mode or Direct SAN transport mode as this configuration is not supported due to limitations of VMware

Information: For more information regarding ‘HostAccessRestrictedToManagementServer’ please see below VMware documentation.

http://pubs.vmware.com/vsphere-55/index.jsp?topic=%2Fcom.vmware.wssdk.apiref.doc%2Fvim.fault.HostAccessRestrictedToManagementServer.html