#1 Global Leader in Data Protection & Ransomware Recovery

Service Provider guide on troubleshooting slow merge in Cloud Connect jobs

KB ID: 2893
Product: Veeam Backup & Replication
Veeam Cloud Connect
Version: 9.5
Published: 2019-02-11
Last Modified: 2020-08-13
mailbox
Get weekly article updates
By subscribing, you are agreeing to have your personal information managed in accordance with the terms of Veeam's Privacy Notice.

Cheers for trusting us with the spot in your mailbox!

Now you’re less likely to miss what’s been brewing in our knowledge base with this weekly digest

error icon

Oops! Something went wrong.

Please try again later.

Challenge

During synthetic operations, data processing is handled solely by a Cloud repository. This article provides troubleshooting recommendations which may help to correctly identify the cause of slowness, increase performance and shorten backup window.

Solution

1) Storage performance

The first thing to determine is a type of the backup chain. Below is the correlation between the backup method and IOPS:

   Method

   IOPS/Block

Active full

1 write I/O

Forward incremental

1 write I/O

Forever forward incremental

2 I/O (1 I/O read + 1 I/O write)

Synthetic full

2 I/O (1 I/O read + 1 I/O write)

Reverse incremental

3 I/O (1 I/O read + 2 I/O write)

Synthetic full with rollbacks

4 I/O (2 I/O read + 2 I/O write)

Synthetic operations cause a significant disk load; thus, it is recommended to monitor IOPS, latency and throughput when processing is performed. For Windows-based repository servers, storage performance can be tested independently with DiskSpd (KB2014).

Synthetic operations can be avoided:

  • in Backup job by configuring it as a Forward Incremental with periodic Active Full backups
  • in Backup Copy job by enabling the option Read the entire restore point from source backup instead of synthesizing it from increments

Note: Active Full strategy imposes a larger amount of data to be transferred between sites.
 

2) Cloud repository server load (RAM/CPU)

Merge puts a significant load on repository server, please make sure that it doesn’t run out of RAM/CPU during backup window.

It is recommended that the task limitation settings are defined using the following rule: 1 concurrently running task = 1 CPU core. For the RAM requirement, if the backup repository server is running multiple Veeam roles, please add up memory requirements of each individual role.

If overload is discovered, consider increasing available resources.
 

3) Max concurrent tasks for Tenant

If Cloud repository is configured as per-VM, the main setting which defines how many VMs in the job can be merged in parallel is the Max concurrent tasks (Tenant Settings > Bandwidth tab). Adding more tasks to a tenant increases the load on the Cloud repository, however, it allows merging more VMs simultaneously, which eventually shortens backup window.
 

4) Antivirus exclusions must be present on all Managed Servers

File-scanning and HIPS activities of antivirus software may significantly slow down or even terminate merge operation. Please consider adding exclusions on Cloud repository servers (KB1999).
 

5) Deduplicating Storage Appliances

As Deduplicating Storage Appliance devices are unable to dedupe encrypted data blocks, synthesized backup file size might be significantly bigger in size. It slows down merge performance and increases backup window size.

More information on limitations can be found here.
 

6) CIFS as a Cloud repository

Under heavy load, with lots of concurrent tasks running towards it, CIFS (SMB) generally offers the poorest merge performance. It may lead to various issues described in KB1735.

The machine which acts as a repository gateway server must be located as close to the backup repository as possible.

If NAS device is used, consider reconfiguring it as an iSCSI target or an NFS mount on a separate Linux server, depending on which type is supported by the device. iSCSI usually offers better performance and stability.

If Dell EMC Data Domain is presented as a CIFS share to Veeam and added as a Cloud Repository, consider switching it to DDBoost to avoid the rehydration penalty during synthetic operations (KB1956).
 

ReFS considerations

For Windows Server 2016 (or later) repositories, formatting volumes into ReFS, will allow utilizing Fast Clone technology, which drastically improves performance of the synthetic operations.

With a ReFS repository in place, mind the following:

  • Latest Windows updates are installed on the repository server.
  • Volumes are formatted into 64K.
  • System requirements for Backup repository servers are met.

 If ReFS performance is still below expectations, please contact support for assistance.

To submit feedback regarding this article, please click this link: Send Article Feedback
To report a typo on this page, highlight the typo with your mouse and press CTRL + Enter.

Spelling error in text

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Thank you!

Thank you!

Your feedback has been received and will be reviewed.

Oops! Something went wrong.

Please try again later.

You have selected too large block!

Please try select less.

KB Feedback/Suggestion

This form is only for KB Feedback/Suggestions, if you need help with the software open a support case

By submitting, you are agreeing to have your personal information managed in accordance with the terms of Veeam's Privacy Notice.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Verify your email to continue your product download
We've sent a verification code to:
  • Incorrect verification code. Please try again.
An email with a verification code was just sent to
Didn't receive the code? Click to resend in sec
Didn't receive the code? Click to resend
Thank you!

Thank you!

Your feedback has been received and will be reviewed.

error icon

Oops! Something went wrong.

Please try again later.