Network backup slow on vSphere ESX 4.0

[sphilp]

[Edited by Gostev] Issue summary: ESX 4.0 service console has disk read cap/throttling issue that results in network backup becoming 3-4 times slower after upgrading from ESX 3.5 to ESX 4.0. This issue is being investigated by VMware. For Veeam Backup, current recommended workaround is to use VCB SAN mode instead (if you are using shared storage). More information can be found in my blog at vNotion.com

Resolution:
ESX patch from VMware is now available

We have upgraded to vSphere (full) and we're seeing a drastic reduction in backup speeds. Where guest backups would previously report 50-70MB/s, we're now seeing things top out at 25MB/s, even for the followup incremental backups.

We struck it up as a fluke on the first datacenter in our main facility, but we're now seeing the exact same performance after installing in our second datacenter.

We're using Veeam Backup 3.1, installed on a fresh Windows 2008 64-bit guest with 2 CPU and 2GB memory. It was previously installed in a Windows 2008 32-bit guest.

The physical server has 8 2.83GHz CPUs, 32GB memory and is showing approximately 20% host utilization at the time backups are occurring.

The hosts are connected via 4GB/s FC to an 8-drive EVA 4400 through a Brocade 200E switch. The backups are writing to a 16-drive RAID50 Samba share. The Samba server isn't experiencing high CPU loads, we're seeing about 6% utilization there during the backups.

The vSphere Service Console is on vSwitch0. The vSwitch has a Service Console port and a VMKernel port. The vSwitch is connected to 2 physical adapters, both Broadcom BCM5708 gig NICs. These are the same NICs that were used under ESX 3.5 and Veeam Backup 3.0.1 with great backup performance.

Have checked Ethernet NICs and switch ports, no errors. Have checked FC ports and switches, nothing odd there either.

The vSphere installations on the two hosts were clean installs, not upgrades. The vCenter install was also on a fresh Windows 2008 64-bit guest. It was previously on a Windows 2003 32-bit guest.

The guests were all updated with the new VMware Tools, part of them were upgraded with the newer VM Hardware. Old or new VM Hardware doesn't seem to make a difference in the backup speed.

Testing the SAN and Samba server via IOmeter get us performance that we'd expect, much higher than we're seeing via Veeam backups.

We have tested backing up via a Samba share (\\server\share) and doing "direct" to the Linux Samba server. Both perform equally as bad.

Have checked the VMware forums for possible issues with the FC hardware, but came up empty. Have watched these forums for the past few days to see if others were seeing this, but didn't see anything there either.

Sort of at the end of my rope as to what is causing this drastic slowdown. A 330GB guest is now taking about 4.5 hours to backup and that's REALLY ridiculous... Sorry for the long post, but wanted to provide the hardware and install steps along with the troubleshooting we've attempted so far.

Would love to hear any suggestions for how to narrows this down and resolve it!

[pjh]

Was your previous W2K8 32bit running with 2 vCPU's? How many other VM's are there running? How many vCPU's are they using?

Cheers!! PJH

[sphilp]

Yes, the previous Veeam Backup guest was also running with 2VCPUs. We've tried both 1 and 2 VCPUs for the guest with no change in performance.

There are a total of 28 guests running on a host in our main facility, they're assigned a total of 36 VCPUs. The secondary facilities host runs 9 guests and they're assigned 19 VCPUs.

In day-to-day operations, only 1 server in each datacenter is needed, so the second host in the cluster is in standby mode. We did test bringing up the second server and putting ONLY the vCenter, Veeam and currently-backing-up guest on the machine. No change in backup speed....

[pjh]

Sounds like the only difference apart from going to vSphere is your VM changing from 32bit to 64bit?

From what I've seen so far Vsphere improves performance.

[sphilp]

Yep, the only changes would be the vCenter and Veeam Backup guests going from 2003 32-bit to 2008 64-bit.

We did test running backups in the original Veeam Backup guest, but had the same dismal speeds there. Since it wasn't that, we went ahead and created the new 2008 64-bit guest and that's what the guest is using now.

We're happy with the stuff we're seeing in vSphere -- memory usage has declined a bunch. Just need to figure out what's happening with the backups...

[lohelle]

[Edited by Gostev]Issue summary: faulty backup storage disk controller (see below for more details and troubleshooting path).

I have tested Veeam Backup 3.0 before against ESX 3.5. Backup speed was 50-80 MB/s (in SAN mode). VCB from command line = 70 - 80 MB/s.

Now in Veeam Backup 3.1 + ESX 4.0, the same SAN and the same backup server/storage backup speed is 15-25 MB/s (optimal compression). VCB is still 70-80 MB/s from command line, but not in Veeam Backup.
The speed is so bad that er are having problems completing backups before the next should start, and that with only 2 TB of data.
The backup server is running Windows Server 2003 SP2 (fully patched) with the newest MS Iscsi Initiator. (32-bit Windows)

[Gostev]

All, have you upgraded VCB to version 1.5 Update 1 that is required for vSphere?

Steve,
What backup mode do you use in Veeam Backup?

Lars,
What command line do you use to test VCB speed?
Do I understand it correctly that you have Veeam Backup installed on physical server, but using software iSCSI initiator?

[sphilp]

All, have you upgraded VCB to version 1.5 Update 1 that is required for vSphere?

Steve,
What backup mode do you use in Veeam Backup?

We do not use a VCB server.

The backup jobs are setup to use "Network backup" (VCB Consolidated backup is greyed out) on the Job properties.

[lohelle]

Using this command when using VCB only (not Veeam)
vcbmounter -h vcenter.domain.local -u administrator -p secretpassword -a ipaddr:212.184.222.222 -r j:\vcb\dc01 -t fullvm -m san

The newest MS SW initiator. Reinstalled the backup server yesterday to check if that helped (did not help)
Single path to Infortrend ISCSI SAN. As mentioned, the manual vcbmounter command gives us ca 80 MB/s backup speed.

[Gostev]

Steve, can you try to copy/paste some large VMDK file of some VM that is not running from your ESX server to Veeam Backup server using Veeam Backup servers tree, and let me know what speed are you getting? Then select "Force service console agent" mode for this server, and do the same thing (just to make sure that you are using service console agent mode).

[Gostev]

Using this command when using VCB only (not Veeam)
vcbmounter -h vcenter.domain.local -u administrator -p secretpassword -a ipaddr:212.184.222.222 -r j:\vcb\dc01 -t fullvm -m san

The newest MS SW initiator. Reinstalled the backup server yesterday to check if that helped (did not help)
Single path to Infortrend ISCSI SAN. As mentioned, the manual vcbmounter command gives us ca 80 MB/s backup speed.

Lars, you are not performing thу VCB speed test correctly, since Veeam Backup configures vcbmounter.exe with different options. You command does not create bit-identical output of VM image as it misses some options. Instead, it creates sparse output, this is what also makes it work faster.

Please, lookup the actual vcbmounter command line in Veeam Backup logs, and try this test again.

Also, I think you missed my question about VCB 1.5 Update 1.

[sphilp]

Steve, can you try to copy/paste some large VMDK file of some VM that is not running from your ESX server to Veeam Backup server using Veeam Backup servers tree, and let me know what speed are you getting? Then select "Force service console agent" mode for this server, and do the same thing (just to make sure that you are using service console agent mode).

Copy / paste: Used a 1.75GB ISO file stored on the same SAN LUN as the VMDK files. The transfer was reported as 18MB/s by the time the transfer finished.

Service Console agent: Turning on the Force flag and running a backup gets us the same 20-25MB/s as we were seeing before. In watching the backups, we're consistently seeing "Backup mode: service console agent", so I don't think anything changed there.

[Gostev]

Steve, ISO is not really good for testing since it does not contain white space. But for instance, for VM with 60% of disk space being free, you should be getting 3 times faster transfer speed than with ISO because white space will be excluded from transfer. This theoretically should provide 50+ MB/s throughtput as you've seen before (and even faster for incremental passes), and we could move on to looking for other bottlenecks.

Could you confirm this by re-testing with some VMDK file?

[sphilp]

Steve, ISO is not really good for testing since it does not contain white space. But for instance, for VM with 60% of disk space being free, you should be getting 3 times faster transfer speed than with ISO because white space will be excluded from transfer. This theoretically should provide 50+ MB/s throughtput as you've seen before (and even faster for incremental passes), and we could move on to looking for other bottlenecks.

Could you confirm this by re-testing with some VMDK file?

Tested with a shut down guest (32GB VMDK, 9GB used, changed to thin provisioned from thick after vSphere installation using svMotion), and I'm seeing 34MB/s at the end of the transfer. We do see the expected increase in backup speed once the "end" of the used space is reached. Prior to reaching that, the transfer was running at around 22MB/s.

[lohelle]

I just tested with the VCB command from Veeam logs. The main difference is the -F 1 -M 1 export flags.
I see that when using the -F 1 parameter the VMDK is preallocated on the backup server (almost no network traffic), and after som time (when passing 50% on the progress bar) network traffic starts (70-80MB/s)
When removing the -F 1 parameter the backup start to "download" at once, and backup is completed very fast.

The backup speed of Veeam Backup 3.0 + ESX 3.5 was about the same as the VCB backup without the -F 1 parameter.

This does not explain Steve's problem I think, but it narrows down mine.

btw: the VM I'm running backup of:
8GB VMDK (thick)
7,5GB data inside