Job fails with error "The create snapshot operation exceeded the time limit for holding off I/O"

Challenge

CreateSnapshot failed, vmRef "vm-xxx", timeout "1800000", snName "VEEAM BACKUP TEMPORARY SNAPSHOT", snDescription "Please do not delete this snapshot. It is being used by Veeam Backup.", memory "False", quiesce "True" Cannot create a quiesced snapshot because the create snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine.

Cannot create a quiesced snapshot because the snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine

Cause

This issue occurs when the I/O in the virtual machine is high and the quiescing operation is unable to flush all the data to disk, while further I/O is created.

This issue also occurs if you try to create a quiesced snapshot on a virtual machine that does not have free space. That is, if the free space on the virtual machine is 0 bytes.

Solution

This error will only occur when the Veeam Backup & Replication job has been configured to enable VMware Tools Quiescence. If this option is enabled Veeam Backup & Replication will use the VMware Tools to freeze the file system and application data on the VM before backup or replication. The option "Enable VMware tools quiescence" is disabled by default. It is strongly recommended to leave it disabled when processing Windows systems. For Windows based systems, it is recommended to use the option "Enable application-aware processing" which will utilize VSS within the guest.

This error can be reproduced by manually creating a snapshot on the VM that is failing to be processed with the option for "Snapshot the virtual machine's memory" unchecked and checking "Quiesce guest file system".

As this error is caused by an issue with the creation of the snapshot on the Virtual Machine by the vSphere Environment, please refer to this KB article from VMware:
https://kb.vmware.com/s/article/1018194