The behavior in question is that when a Windows Server 2008 or Windows 7 virtual machine is cloned or otherwise incurs a change to the MAC address; the virtual machine will re-enumerate the Ethernet interface. This also happens when using Veeam Backup and Replication’s SureBackup functionality. While I’ve historically been a fan of the VMXNET3 interface, everyone should be aware of this behavior. There are limitations with VMXNET3, such as it not being supported for use in the VMware Fault Tolerance (FT) virtual machine configuration for all versions of vSphere.

The re-enumeration of the Ethernet interface will make it appear still as a VMXNET3 interface, but in the operating system; it will become the next in sequence. For a virtual machine that is created with the default option of selecting an automatically assigned MAC address, the network interface will show up as shown in the figure below if the MAC address changes:

This is all too familiar as many virtualization professionals had this happen when performing physical to virtual conversions, as well as the upgrade that we may have performed going to VMXNET3 from a previous adapter type. Because of this behavior of the adapter, VMware has published this KB article recommending the use of the E1000 adapter type for templates using Windows Server 2008 or Windows 7. If the virtual machine has a custom-defined MAC address, most of these issues do not occur; but this isn’t really a practical solution.

This is just the case when a virtual machine that is launched within SureBackup has an automatically configured MAC address, the virtual machine may not respond correctly to this new environment. The primary observation in most situations is that the guest virtual machine will enumerate the additional network interface (as shown above) and it will not retain any of the networking configuration options that were part of the source virtual machine. This means that the interface will boot up requesting a DHCP address and not have the static IP address configuration that was previously assigned. This doesn’t apply to the other virtual network adapter types (such as E1000 or VMXNET2) or other operating systems such as Windows Server 2003.

For the same virtual machine that was shown above, it receives a new MAC address in the automatic configuration. This is due to how the VMXNET3 interface enumerates itself in Windows. The VMXNET3 device shows its enumeration in the Windows Device Manager as shown below:

This corresponds to its location in the NetworkCards hive of the registry. Basically, each time a new network card interface is enumerated in the operating system; they are displayed here as well. Here is where these IDs are enumerated in the registry:

While all of this is the rather standard experience that we have gone through in cloning a virtual machine or related tasks, it can impact the SureBackup feature of Veeam Backup and Replication 5. In this forum thread, a Veeam user highlighted a situation where using VMXNET3 can impact the SureBackup functionality. Anton Gostev, a product manager at Veeam, points out that the 5.0.1 release of Veeam Backup and Replication will have a workaround.

Update: Version 5.0.1 has been released. Be sure to update previous versions to the latest release.

Since the release of Veeam Backup & Replication v5 in October, we have gotten great reviews and invaluable feedback. You can read Eric Siebert’s review of Veeam Backup & Replication v5 that was posted on Searchdatabackup.com or see the in-depth review posted on Vladan.fr.

As you probably already know, Veeam Backup & Replication has features that are not available in any other existing products on the market:

• Instant VM Recovery: Restore entire virtual machines from a backup file in minutes.
• U-AIR (Universal Application-Item Recovery): Recover individual items from any virtualized application, on any OS.
• SureBackup Recovery Verification: Automatically verify the recoverability of EVERY backup, EVERY VM, EVERY time.
• On-Demand Sandbox: Create test VMs from any point in time to troubleshoot problems or test workarounds, software patches or new application code.
• Instant File-Level Recovery: Recover an entire VM or an individual file from the same image-level backup in any OS or file system.

With this promotion you can get 25% off Veeam Backup & Replication v5 when you provide a proof of license for a competing product. This is a limited time offer that will end soon. If you are not satisfied with your existing backup solution, give Veeam Backup & Replication v5 a try.

The second promotion that will end right before Christmas is on Veeam Essentials Plus. You can save 42% by getting a free upgrade to Veeam Essentials Plus when buying Veeam Essentials.

We are very glad to be a part of the “IF ONLY virtualization was…” campaign with VMware. VMware Essentials Plus and Veeam Essentials Plus will give SMB’s the ability to manage and protect their virtual infrastructure at an affordable price.

Veeam Essentials Plus includes Veeam Backup & Replication v5 Enterprise Edition (features listed above), the industry standard for VMware monitoring in Veeam Monitor 5.0, Veeam Reporter, Veeam FastSCP and Veeam Business View.

Learn more about this VMware and Veeam promotion.

Take advantage of these two great promotions before they are gone.

One of the points I liked about the Storage Switzerland article was the entire history aspect of backup vendors…

In almost every case, when a new operating system or platform became dominant, a new data protection leader emerged. Novell NetWare begat Cheyenne Software, but as UNIX and Microsoft Windows NT emerged, this NetWare solution couldn’t make the transition to the new platforms. UNIX backup drove the establishment of Legato NetWorker (eventually EMC NetWorker) and Openvision NetBackup (eventually Symantec NetBackup). Microsoft Windows NT drove the establishment of Arcadia Backup Exec (eventually Symantec Backup Exec). Even CommVault, one of the more successful newer backup applications, was driven by the establishment of Windows as an enterprise, mission-critical operating system. In short, each new operating system or platform eventually brought a new data protection application.

This truly does show that it’s a great time to be a vendor focusing on the VMware platform (and virtualization in general). Expect more great things from Veeam as we continue to innovate on these virtualization platforms.

From the Microsoft perspective, there are 3 core rules to backup and recovery of Exchange servers:

In order to be compliant with Exchange Server, VSS based backup applications must follow three basic requirements to ensure the integrity and recoverability of shadow copy backups. If these requirements are not followed, Microsoft ... will not be able to troubleshoot backup and restore issues.

Rule 1: Exchange must be backed up exclusively through the Exchange VSS Writer.

Rule 2: Backup should not be relied on until the backup application has completed integrity verification.

Rule 3: Restores to original location must be done exclusively with the Exchange VSS Writer.

Rule 1: VSS Aware Backup

Veeam implements proprietary Microsoft VSS integration, instead of relying on VMware Tools VSS integration components.

• Fully automated and transparent (no agents to deploy/configure/update/monitor)
• Supported directly by Veeam, not VMware (no finger pointing)
• No limitations of VMware Tools VSS: supports transaction logs processing, all ESX(i) and Windows versions, dynamic disks, IDE disks, VM without UUID, etc.

More information:

• Veeam Backup v5 FAQ: http://tinyurl.com/v5FAQ
• VMware VSS limitations: http://www.sys-con.com/node/1544145

Rule 2: Verify Before You Rely

SureBackup Recovery Verification

• Great flexibility (supports custom scripts)
• Choose method of verification that is sufficient for you: remote run eseutil or isinteg on test VM (no stress on production), log on to test mailbox via HTTPS and query test email message

Keep in mind DC dependency!

• Exchange must see DC to be able to properly boot in the isolated environment. SureBackup Application Groups take care of this for you.

Rule 3: VSS Aware Restore

Restores to original location must be done exclusively with the Exchange VSS Writer, and in correct sequence:

• Boot up Exchange VM with mailbox stores dismounted
• Tell Exchange VSS Writer to perform restore from VSS snapshot
• Mount mailbox stores

Veeam implements these Microsoft requirements

• Most image-level backup vendors do not do this, they just boot VM normally like there is no Exchange present
• Perform a test restore to check your current solution and look for these events on the restored Exchange server, if they don’t exist your vendor is not following Rule 3:

Transaction Logs

If transaction log files are not pruned after backup, the log files accumulate until they fill all the available disk space. The Exchange VSS Writer implements transaction log pruning capabilities, however VMware Tools VSS is NOT a backup application and cannot know if backup was completed successfully. Thus, it cannot process transaction logs by design.

• Any application “riding” on VMware Tools VSS instead of providing proprietary VSS integration will not truncate logs.
• Some solutions do provide transaction log pruning, but perform log pruning right after the snapshot is taken.

This approach is actually worse than no pruning at all: if backup does not complete successfully, you will not have a good backup, and your transaction logs will be gone. You will not be able to restore in case of disaster.

To check your current image-level solution, perform test a backup to check (on a test Exchange server, not production)

• Perform backup, wait for the job to complete successfully, ensure transaction logs are actually pruned.
• Perform another backup, but this time reset the backup server while the job is running (after virtual disk copy starts). Transaction logs should NOT be pruned.

Veeam prunes logs on successful backup by default and v5 provides advanced transaction log handling options as seen in this screen shot:

Granular Recovery Challenges

Typically granular recovery from an image-level backup was difficult, you had to restore entire Active Directory and Exchange servers to an isolated environment before your could restore any items. The process is time and personnel resource intensive. There are some 3rd party tools that mount the Exchange data store but these still require data stores to be extracted first (time and disk space) and there’s an additional licensing cost associated (usually per mailbox)

Agent-based solutions have existed for years that can back up the Exchange data, but that’s not the most efficient way to backup Exchange in a virtual environment. Additionally, if you combine agent based with image based, you are backing up the same data twice, taking additional resources and storage media.

Granular Recovery with vPower™

Veeam’s patent-pending approach fully utilizes the existing virtual infrastructure. The Veeam application group and virtual lab features automatically create an isolated environment and with vPower, you simply run the AD and Exchange servers directly from the backup files, no extraction necessary.

Veeam’s Exchange AIR (Application Item Recovery) Wizard utilizes Microsoft Exchange APIs and connects to both the production and isolated environments providing you with Exchange item-level recovery in minutes, not hours!

More Information

Microsoft on Exchange 2003 VSS Backup and Restore http://support.microsoft.com/kb/822896

Microsoft on Exchange 2007 VSS Backup and Restore http://technet.microsoft.com/en-us/library/dd233256(EXCHG.80).aspx

Application-aware image processing section of v5 FAQ http://tinyurl.com/v5FAQ

OpenBench Lab Report: Better Data Protection for VMware vSphere

By Jack Fegreus Managing Director openBench Labs.

System and storage virtualization introduces multiple levels of logical abstraction and resource redirection that can obscure and complicate important IT operations. Among the hardest hit IT operations are those associated with file-level data protection.
Read this white paper by industry expert Jack Fegreus to learn how you can:

• Minimize vSphere disk risk with backup and replication - combine backup and near-CDP-level replication to unify both data protection processes for IT with a synthetic backup process and consistent process wizards.
• Maximize backup performance with deduplication and compression - how you can reduce the footprint of full VM’s backups by 3 – 9 times with data deduplication, compression, and recognition of thin provisioning.
• Leverage full support for ESX and ESXi without VCB – fully utilize the new vStorage API or VMware Consolidated Backup (VCB) when performing backup or replication, allowing SMB sites with or without shared storage to fully leverage ESXi, including the ability to replicate VMs to an ESXi host.
• Dramatically increase full and incremental backup speed and reduces storage requirements by utilizing new vSphere Changed-Block Tracking and thin provisioning.
• Higher RPO support via Virtual CDP with replication - leverage inline data deduplication and compression to provide near continuous data protection (CDP) with replication.
• Ensure transaction consistency at VM and application levels - ensure VM transaction consistency and Volume Shadow Services (VSS) for VSS-aware Windows applications such as Active Directory, SQL Server, and Exchange.

Download this white paper and learn how to ensure better VMware vSphere data protection through unified backup and replication.

http://www.veeam.com/go/better-data-protection-vsphere

5 Ways VMware vSphere Improves Backup and Recovery

By Eric Siebert

VMware vSphere™, the industry’s first cloud operating system, leverages the power of virtualization to transform datacenters into dramatically simplified cloud-computing infrastructures. VMware vSphere includes many new features and technological enhancements that can benefit you. While there are several reasons to upgrade to vSphere, this white paper focuses on how VMware vSphere and its vStorage technologies improve data protection and disaster recovery.

Read this white paper by industry expert Eric Siebert to learn how you can leverage vSphere to achieve:

• 30% average reduction in storage space
• 50% average increase in backup speeds when performing full backup
• 90% average increase in backup speeds when performing incremental backups
• 80% lower cost for near continuous data protection (CDP)
• Increased reliability

About the Author
Eric Siebert is a 25-year IT veteran whose primary focus is VMware virtualization and Windows server administration. He is one of the 300 VMware vExperts for 2009, author of the book “VI3 Implementation and Administration” and a frequent TechTarget contributor. In addition, he maintains vSphere-land.com, a VMware information site.

http://www.veeam.com/go/vmware-vsphere-backup-wp

How Virtualization Cuts CDP Costs by 80%

By David Davis Director of Infrastructure at TrainSignal.com

Until recently, CDP has been used only by the largest enterprise companies and only for their most critical data. For small and medium businesses (SMBs)—it was simply cost-prohibitive.
Thanks to virtualization, advanced disaster recovery methods that result in fast recovery and low system downtime are now available to all organizations, from the largest enterprise to the smallest business, for a very reasonable cost.
This white paper compares 3 different options of implementing CDP:

• "Legacy CDP" or Classic SAN-based CDP
• Software-based Server Replication
• Virtualization-enabled "Near-CDP"

Download this white paper and learn how to ensure business continuity and enable disaster recovery at an affordable price

http://www.veeam.com/go/vmware_vsphere_continuous_data_protection

So if you’re going to be at PEX, be sure to stop by the Veeam booth. Also, feel free to find me on Twitter as I’m sure someone will be organizing a TweetUp or 2.

So why do we still have support for VCB? The simple answer is that we have a number of current customers that are using VCB and have not switched over to vSphere yet. We could have dropped support for VCB but then that would require us to maintain both 3.x and 4.x releases. Since 4.0 is an built upon 3.x and not a complete re-write of the software, we left VCB support in for our customers. The benefit is that our customers have a choice of how to process their backups: vStorage APIs, VCB or Veeam’s Service Console or Network modes. Of course I do recommend that customers use the vStorage API methods, it gives you the best performance and is also built specifically for vSphere.

Up to 10x Faster

Yes, it’s really true and our customers have posted as much on our forums. We’ve consolidated this feedback into a handy PDF document INSANE BACKUP SPEED, it’s not us saying this, it’s real people using our 4.0 product.

How is it so fast? One of the biggest reasons is Changed Block Tracking (CBT) which is included in the Virtual Disk Development Kit (VDDK). With CBT enabled for VM’s running on ESX or ESXi 4.x, VMware actually tracks the block level changes made to the VMDK. This way, when software like Veeam Backup & Replication takes a Snapshot, VMware returns a list of what blocks have changed. This significantly increases backup speed as we no longer have to try and determine the changes ourselves, VMware tells us in a matter of seconds.

The Critics

Of course anytime a vendor releases software that’s cutting edge and ahead of the rest of the field, others will try and knock it down. This recently happened on a blog sponsored by one of our competitors, claiming Veeam Backup & Replication was corrupting data. Of course when I first saw this I was a bit alarmed, but since they included a link to our forum where a user was reporting the problem, I followed the link and realized they were just sensationalizing something for their own apparent gain. Since this particular blog decides to heavily moderate comments, I thought I would post the facts here, lest any other competitors decide they want to sensationalize this non-issue.

The Facts

Taken from the Veeam Forum post:

1. Backups are NOT corrupted.
2. You can only run into this issue with NON-DEFAULT restore mode, in 1 restore mode of 3 existing modes.
3. Despite what competition may be claiming, there is no actual user data loss or corruption - VM will still boot and work.

The only real issue is OS and file system check tools complaining about unexpected content of the unused disk blocks. Linux ext3 file system and disk test tools merely suspect a problem seeing unused blocks being non-zeroed, and warn about this. This is specific to certain file systems only, for example, Windows NTFS considers this situation absolutely normal.

Of course every software has bugs, the example above is a bug in our software when using 1 particular recovery method and we have a fix available and it will be included in our next release. The important piece of information is that NO DATA WAS EVER CORRUPTED, just an issue on recovery and zero byte blocks. We are very thankful to our active forum community and for bringing this to our attention. Veeam’s motto after all is “Listening to You, Building the Tools You Need”, it’s on the back of all of our business cards. I’d like to point out the below points about our user forum:

• Our forum is one of our greatest resources where people share their experience, best practices and getting support from Veeam and the community
• We understand that our competitors are so desperate that they will continue using our own forum to try and fool the community
• We will continue to be honest and direct with our customers in our forum vs. ending all the threads with ‘please contact our support’
• User feedback: The quality and effectiveness of this Forum alone would be enough to justify switching to Veeam, even if the product wasn't superior to its competition (as it is at the moment). It seems just too good to be true... I hope it will continue this way!

Conclusion

Sorry to waste so much space on this post answering the competition. If they would allow comments on their own blog I would have written it there. In closing I just want to say that we encourage everyone to evaluate our software and make decisions for themselves. Put us in the lab and you will see, others promise, Veeam delivers!

Check out NetEx's press release here and also check out their YouTube video below. You can download a 10 day free trial of HyperIP and also a free trial of Veeam Backup & Replication to test it out on your own.

You can follow both @HyperIP and @Veeam on Twitter.

To become a VCP VI3 you need to do three things:

1. You have to take the VMware authorized course. No if, ands, or buts! The classes are instructor led and will allow you to get your hands dirty with VMware. There's three courses you can take.

• VMware Infrastructure 3: Install and Configure V3.5
• VMware Infrastructure 3: Deploy, Secure and Analyze V3.5
• VMware Infrastructure 3: Fast Track V3.5

2. Get hands on experience with VMware. What's the point of trying to get a VCP without using it!?

3.Sign up and pass the exam: http://www.pearsonvue.com/vmware

Now that doesn't sound too hard, does it? Well, I wouldn't say taking IT exams is easy. (Unless, you're dumping, which I completely do not agree with.) Usually the hard part of getting a VCP is money. $2500-$3000 a course seems a little steep for a lot of people.(Although, Fast Track = $5495 but includes the exam) I see where VMware is coming from, especially due to the current state of IT certification. (i.e. people cheating/dumping exams.) Also, with the course you receive a VMware messenger bag, I mean, that's totally worth $3000, right?

After you have passed your exam, you will receive a complimentary license of VMware workstation, a $175 "savings". Interesting that the course itself costs about the same.

So, why get your VCP? Why does anyone get IT certifications? We all have different reasons, career advancement, your employer requires you to, or you just like to learn. Whatever your reason may be, getting certified in a technology is always beneficial.

Below are some resources you can use to learn more about VMware certifications:

VMware Education Services

TechExams.net Virtualization Forum

The Value of the VCP (VMware Certified Professional)

So, for just shy of 1K you get fully licensed vSphere. This is great but it still leaves you without an efficient backup and recovery tool, single pane of glass monitoring or detailed reporting...enter Veeam Essentials bundle...offering Veeam Backup & Replication, Veeam Monitor and Veeam Reporter for around US $2,000 (Americas pricing). The Veeam Essentials bundle is fully functional licensed versions of those 3 products for 6 sockets (3 hosts).

Of course Veeam is placing some limits on this bundle...

• Must be purchased with VMware vSphere Essentials (or proof of VMware vSphere Essentials must be given)
• Each bundle will only support 6 sockets. This means that even if you purchase multiple bundles, each Veeam product will only work on 6 sockets, you'll need to install other instances with the other license(s) to manage the other VMware vSphere Essentials vCenters.
• Each bundle is 6 sockets, no less (sorry folks, you can't get it cheaper for only 4 sockets)

What does all this mean? For just under US $3,000 (hardware not included) you get 6 sockets of VMware vSphere, 6 sockets of Veeam Backup & Replication, 6 sockets of Veeam Monitor and 6 sockets of Veeam Reporter. That's 3 servers with up to 8 cores (2x4) and 256GB RAM each running VMware ESX(i). Couple that with a free iSCSI solution such as Starwind and now you're talking 2 TB of shared data for free (hardware and Windows licensing not included).

I did not include Support and Subscription (SnS) above because for VMware vSphere Essentials it's optional (it's not optional on VMware vSphere Essentials Plus). Veeam's Essentials bundle includes 1 year of support and upgrades in the list price.

Now, let's discuss a few things about the Free Version of ESXi. We've gotten a number of questions on this since we released the "news" and the Essentials bundle on the same day. Below is a simple table explaining what is and isn't supported:

Table of compatibility:

+ = supported/usable
- = not supported

Now please note that Veeam will not be supporting ESXi Free 4.0 at all. If you're a current customer using Veeam Backup & Replication for ESXi Free 3.x, please contact your Veeam ProPartner or sales person for vSphere options.

While I don't endorse or condone any of the following, I thought I would be doing an injustice if I did not include links to the community's reaction to our announcement regarding dropping ESXi Free support. The list below is just some of the reaction, be sure to read the comments in each as well.

Rick Vanover at Virtualization Review

Rich Brambley at VMETC

Eric Scholten at VMGURU.NL

Gabes Virtual World

Search Server Virtualization's Alex Barrett

Rich Brambley at VMETC again after Alex Barrett's post

Search Server Virtualization's Eric Siebert

Edward L Haletky's The "soon to be launched" Virtualization Practice

The comments to my post on the original announcement

Update: June 12, 2009 - New Links

Bridget Bothelo over at Virtualization Pro

Discussion over at ars technica (I noticed this in my referral stats)

Update: June 15, 2009 - New Links

Search Server Virtualization's weekly Podcast: VMware cuts free ESXi hypervisor support, fuels battle vs. Hyper-V

Update: June 26, 2009 - New Link

Mike Stokes over at eGroup

Update: July 3, 2009 - New Link

Tom Howarth over at PlanetVM

Mike Laverick from RTFM