Backup Copy or Replication job using WAN accelerators fails with “Source WAN accelerator error: Failed to decompress LZ4 block”

Backup Copy or Replication job is failing with errors such as, or similar to, the following:
 
Error: Source WAN accelerator error: Failed to decompress LZ4 block: Bad crc
 
Error: Source WAN accelerator error: Failed to decompress LZ4 block: Incorrect decompression result or length
 
Error: Source WAN accelerator error: Client error: Zlib decompression error: [-3].
 

Cause:    Cyclic Redundancy Check (CRC) failed while extracting data from a backup file indicates that one or more blocks in that file are corrupted.

Typically, this error indicates hardware malfunction during storage or transmission of the backup file. Most types of storage are subject to a small degree of unrecoverable failure – this is commonly called Bit Rot.

In a small sample of cases this has been seen to be caused by corruption in the WAN accelerator Cache itself rather than the backup file.

 

To isolate, start by configuring the backup copy/replication job to use the direct transport method temporarily disabling WAN acceleration.

If the job works after disabling WAN acceleration, the problem is with the WAN accelerator cache and creating a new volume/rebuilding the WAN accelerator cache may be the next step.

If the error persists (typically seen only in a backup copy, not replication) it means the problem most likely exists with backup files not being able to be recompressed. In this case:

2. Run an active full backup on the source job.

The backup file will be completely recreated from source data. Once done, re-run the backup copy job and it will grab the data from this backup file and should bypass the error. If not:

3. Trigger an active full backup on the backup copy job.

Veeam will read the data entirely from the new backup file, bypassing all previous backup files on both the source and the target repository. This will isolate the problem from any previous backup file corruption. Should the error persist at that point:

4. Run a chkdsk or equivalent function on both the source storage, the VM datastore storage, and the target storage for this backup copy job. We’ve recreated, at this point, source and target backup files and ruled out the WAN accelerator components, so as uncommon as it is, it’s likely corruption on one of the storage volumes causing consistency errors.

This is the same as the errors mentioned in the below KB:
https://www.veeam.com/kb1795

 

The Windows event log on the repository server may contain Disk Event 7: “The device… has a bad block.” If so, run CHKDSK /F or /R. A small number of bad blocks are normal on most storage devices. If additional bad block events occur after running CHKDSK, check the SMART status and consider replacing the disk. If using a Network Attached Storage device, contact your storage vendor to determine what diagnostics are available.

It is strongly recommended that you test your backup files onsite before sending them to an offsite location for storage.  You can do this through test restores or by using the SureBackup feature included in Enterprise or Enterprise Plus licensing. For more information on SureBackup see the below links:

Practical Guide to SureBackup
Veeam Backup & Replication - SureBackup Step-by-Step Configuration Guide