The Great VSS Debate

This morning my inbox greeted me with an email from Eric Siebert and my Twitter search folder filled with statements about Veeam’s VSS claims. After my first cup of coffee I started to dig in and see what in the virtual world was going on…this first lead me to a blog post by EMC’s Scott Waterhouse where he states the following:

Therefore, final answer: if you need application consistent backups, you must do a guest level backup. An image level backup is simply not good enough. Even with a MS Windows 2003 or higher guest, even though VMware supports VSS. Yes you may do image level backups too, but they will only complement, no replace, guest level backups.

To be honest Scott was talking about VMware VSS here. I still don’t agree with his statement that for application consistent backups you have to do a guest level backup because there ARE solutions on the market today that provide application consistent backups from an image…of course that would be Veeam Backup & Replication.

What prompted some of the discussion (as far as I can tell) is that people don’t realize that Veeam uses its own VSS driver, not VMware’s. This has been the case since Veeam Backup and Replication 2.0, released in July of 2008. In fact, at the time, we pointed out how our VSS integration was different from the competition through a couple of blog posts: Is your backup really VSS aware? and VSS and VMware ESX: What your VMware backup vendor isn’t telling you. These posts really talk about VSS recovery and making sure that your VSS “backup” is truly able to recover properly. These 2 posts continue to be very popular on this blog…maybe pointing out that there is confusion regarding proper VSS handling when it comes to VMware.

That was all in 2008, what’s going on with VSS in 2010? Well, Veeam has continued to update its VSS integration to keep pace with the new releases of Windows Server 2008, both x86 and x64. Not all vendors have been keeping up with Microsoft though, in fact, looking at VMware’s VCB/VSS support table, it clearly states that they don’t support application consistency on Windows Server 2008, only file-level. So, any virtualization backup vendor that relies on VMware’s SYNC Driver and VSS for file and application consistency is not giving true VSS support for Windows Server 2008 (as Scott pointed out in his blog).

I’d like to finish up by answering Scott’s call for documentation:

Unfortunately, nobody has been able to provide a piece of definitive technical documentation (a white paper, or a support document, or relevant piece of text from an administration guide) that clearly describes the issue.

First, we have real users, using Veeam’s VSS today, that are talking about it in our forums, this is proof that they’re getting application consistent backups from Veeam’s image level process using our VSS:

Post by tsightler » 05 Jan 2010 14:22

Once again, Veeam fully support VSS aware snapshots of both AD and Exchange server when using the Veeam VSS Agent. Veeam doesn’t just “take a VM copy”, the Veeam VSS agent uses Windows VSS services to put these features into a proper, supported VSS backup state prior to taking the VM snapshot. In other words, a Veeam backup is indeed a “backup-aware copy of the Information Store and NTDS database”, and it uses the Windows recommended VSS processes to achieve this.

Answer by donikatz » 06 Jan 2010 00:00

Obviously neither Tom nor Anton need my help here, but maybe some real-world testimony would make you feel more comfortable? Not only have I tested this, I’ve performed a *production* restore of a w2k3 DC with Veeam and it worked exactly and as simply as in the video. I’ve also done several *production* SQL restores without issue. Veeam also works well in our Exchange restore tests, although we haven’t had to do any in production (knock on wood). Although agent-based apps like Backup Exec may have more direct hooks for simpler granular restore (we still use BE for Exchange brick-level restores because our admins are more familiar with the process), Veeam is more than capable without the drawbacks of an agent. I hope to move away from BE altogether for Exchange this year; it’s just a matter of updating our runbook and training. Honestly, if there’s one area you certainly don’t need to lose sleep over with Veeam, it’s with Microsoft products. MS has well-proven APIs and Veeam makes great use of them; Veeam VSS is excellent. Heck, if only Oracle on Linux had VSS the way it does on Windows it would make my life a lot easier…

Next, I’ve taking some quotes from Veeam’s own user guide regarding the differences between using VMware tools quiescence (SYNC) and Veeam’s VSS driver:

Transactionally Consistent Backup

Veeam Backup & Replication 4.0 provides two techniques for creating transactionally consistent backup images — the Enable VMware tools quiescence and Enable Veeam VSS integration options. In contrast to restoring a crash-consistent backup, which is essentially equivalent to rebooting a server after a hard reset, restoring transactionally consistent backups ensures safety of data of applications running on VMs.
Please note that when you select both VSS integration and VMware tools quiescence options for a job at the same time, the VSS module will only be used for processing backed up and replicated VMs. However, if you use both VSS and VMware tools quiescence options and select the Continue backup even if Veeam VSS quiescence fails option for backup jobs or the Continue replication even if Veeam VSS quiescence fails option for replication jobs, all your VMs will be processed with VSS first, and in case of VSS failure (e.g., Linux VMs), VMs will be processed with the VMware tools quiescence option enabled.
This can be very useful when you have both Windows- and Linux-based VMs in one job, so all VMs will be processed in a transactionally consistent way using VSS or VMware tools quiescence option.

Additionally, we then go on to explain the VSS process as well as the systems supported by our VSS driver:

Enable VSS Integration

With the Enable VSS integration option selected, Veeam Backup & Replication 4.0 utilizes the Windows Volume Shadow Copy Service (VSS) that ensures consistent backup of VSS-aware application running within your virtual machines (domain controllers, databases and other applications) without shutting them down. The Enable Veeam VSS integration option allows creating a transactionally consistent backup image of a VM, which, in contrast to a crash-consistent backup image, ensures successful VM recovery, as well as proper recovery of all applications installed on the VM without any data loss.
In the process of its work, VSS freezes all I/O at a specific point-in-time by interfacing with all VSS-aware applications and the Windows operating system. Consequently, there remain no unfinished database transactions or incomplete application files. Such backups, when restored correctly, result in fully functional applications.
The VSS works with Windows 2003, Windows XP, Windows 2008, Windows 2008 R2 and Windows 7 guest operating systems. Use VSS to back up 32-bit or 64-bit version of Windows 2003, 32-bit version of Windows XP guest OS, 32-bit and 64-bit versions of Windows 2008. Please note that administrator credentials are required to access the guest OS. Microsoft Windows VSS backup option requires that your guest OS has VMware Tools, and all the latest service packs and patches installed.

If anyone has any questions or further thoughts I’d love to hear from you. Feel free to comment below, hit me up on Twitter or shoot me an email @ doug /dot/ hazelman /at/ Veeam /dot/ com

Get weekly blog updates
By subscribing, you are agreeing to have your personal information managed in accordance with the terms of Veeam’s Privacy Policy
Cheers for trusting us with the spot in your mailbox!
Now you’re less likely to miss what’s been brewing in our blog with this weekly digest.

Eliminate Data Loss
Eliminate Ransomware

#1 Backup and Recovery


  • Setting aside the who does what discussion, my original post(s) were really trying to establish what is possible with VMware + VM Tools + vSphere 4.0. My premise was simply that VM Tools may have limitations–and what were those limitations? The other thing that I think is relevant is that, given a preference, and all other things being equal, I would rather use VM Tools than install another agent (Veeam VSS). Having said that, either approach is superior to guest level backup. But (all) I was doing was contrasting the native VMware Tools + .vmdk image level with guest level backup. Frankly installing your own agent is cheating to a certain extent :) and outside the scope of my original discussion. As I put in my tweet (image level + VM Tools > image level + 3rd party agent) > guest level. Finally, we all love VMware and I am not going to throw any stones at anybody until I have a clear, definitive, documented answer from them.

  • vmdoug says:


    No worries, I just wanted to let everyone know that Avamar is not the only solution, as you alluded to in your post.

    As for our “agent”, it’s completely run-time, no need to pre-install or re-install after an update…just supply login credentials when you setup the Backup or Replication job and Veeam handles the rest. Why call it cheating when it makes the process actually work?

    Also, isn’t this table enough documentation from VMware?

  • Brendan says:

    So Veeam does not require persistent agent for VSS? Impressive… we currently use Symantec and it does require one.

    I agree, biggest issue with agents is having to deploy, update and watch those behave. Also, not having any 3rd party bits installed and running on VM most of the time is nice added benefit I guess :)

  • That documentation doesn’t mention applications (Oracle, SQL, Exchange, AD, etc.) and doesn’t talk about ESX 4.0 and vSphere–if I am interpreting it correctly it is just ESX 3.5 + VCB. So no, it isn’t enough!

    As far as the “cheating” comment–I guess I look at it from two perspectives. One, I can think of ways of doing this that would not even require VMware Tools. So anything else is not really what I want (from a blue sky long term perspective) and therefore cheating. Secondly, from a more real world today perspective, I would want to know more about the integration with vSphere: does it auto-detect and install on new VMs? How about maintenance? Upgrades? Protection monitoring. That sort of thing. Maybe you have good answers, but I would definitely want to understand the pros/cons before I soften my “cheating” response.

  • The doc doesn’t mention apps by name, but it does mention that they can do application quiescieng for W2003, not W2008. BUT, as you pointed out, it only applies to VCB not to vSphere. And VCB is SOOO last decade.

    As to the cheating part, their agent is no more cheating than your agent. You said there’s no way to take application consistent backups without doing host-based backups — and that the best host-based backup for VMware was Avamar (paraphrasing you). Clearly there is a way to do it without doing host-based backups, and it’s Veeam.

    Had you said “there’s no way to do it without putting some kind of client on the host,” you might be closer to the truth. Although that one doc does seem to suggest that VMware at least understands the difference. What we don’t have is a table that shows what they support in vSphere and we’re assuming that if they do application VSS, they do all applications that have VSS. If only there was a compatibility matrix….

  • A compatibility matrix? :) What a concept! From the get go that is exactly what I have been asking myself. Why isn’t there a compatibility matrix that lists OS versions, apps, VM versions and spells it out clear for the dummies like me in the audience?

  • vmdoug says:

    With VCB VMware was trying to provide a backup framework, it makes sense that they included VSS support since it was a customer scriptable solution. With the vStorage API’s for Data Protection (vShpere), they are asking the backup vendors to hook into the API’s rather than requiring an existing framework (VCB). This is a great move on VMware’s part, but is it their responsibility to provide complete VSS support or is it the backup vendor’s responsibility? I would say it’s the backup vendors that need to provide the VSS support with vSphere, not VMware. VMware could do a better VSS job with their vDR product, but that’s a backup product, not a set of API’s for backup vendors to use.

  • vmdoug says:


    Correct, Veeam’s VSS “agent” is not persistent, it’s run-time only so you don’t have to worry about a separate install, the backup or replication job installs the VSS executable, then removes it once the backup is complete. When we release a new version, we update the VSS executable but there’s nothing that the end user needs to do since it’s not persistent.

  • Barry Coombs says:

    It’s good to know the levels to which Veeam is protecting our VMs but for me we still have to run agents for exchange / sharepoint / ad etc, for the granular restore ability. Now a virtualised backup product that would allow this from an image level backup would be worth it’s weight in gold! But nice to know if I did need to restore a complete VM the likleyhood is such DB’s would be ok due to Veeams VSS driver.

  • rickyelqasem says:

    Barry you can perform granular restores of Exchange, SQL and so from an image-level backup if you know what you are doing. For example a single email restore. Requires some manual steps in the middle but negates the need for expensive agents. OK I admit not as slick as having the functionality built-in to the GUI but it can still be done at no extra cost.

  • Barry Coombs says:

    Customers appreciate the speed and ease from their agents on these kind of applications, so they tend to keep there agents from the physical world when they virtualise. Just would be nice to take it down to one backup product in use.

    If a vendor ever came out with a solution that meant we could manage the whole backup process with 1 piece of software it would be a high priority to investigate.

    At the moment it tends to be virtualisation backup vendor (Veeam usually) and Traditional vendor (backup exec usually) for agents and to tape portion if needed.

  • What this blog fails to mention is that Veeam only offers “application consistency” and “transaction consistency” for backups in VMs *if* your application suite stores information on a single server.

    This is not only our own experience during our recent evaluation of Veeam, but also straight from the horse’s mouth – a Veeam engineer named Mikhail who sent me a somewhat condescending email.

    When you have the requirement of having to take snapshots to backup systems that distribute their transactional information across multiple servers – such as Datasets with HP’s Trim eDRMS (which stores files on a regular file server and meta-data in an RDBMS) – then you are SOL with Veeam.

    I wish it weren’t so – we liked Veeam and loved its performance, lack of needing a client agent, integration of both backup and replication in a single suite. But there’s hype and there’s the reality of real-word application suites.

  • vmdoug says:


    Yes, it’s very difficult to get application consistency across an application that stores transactional data across several servers. Since Veeam uses VMware snapshots to create backups, we don’t have a way to control exactly when each snap gets created. For applications like you suggest (that store transactional data across several VM’s) the only true way to get a consistent backup would be to use agents on the VM’s that are aware of that particular application.

    The intent of the post was not to say that Veeam can backup EVERY application by using VSS (the application itself has to be VSS aware) but simply state that Veeam’s VSS implementation is more complete than others out there, including VMware which is not application VSS aware on Windows Server 2008.

  • Brendan says:

    To be fair to Veeam, I don’t see how this could possibly be implemented without completely suspending the whole system. Basically, it is technically impossible to create snapshots of multiple VMs at once. So in your example, hot backups will always result in differences between metadata and files state (from seconds to minutes).

Leave a Reply

Your email address will not be published.