We have repeatedly asked Tolly who is sponsoring the report but they have declined to tell us. As of this writing, Veeam has no idea who is paying Tolly for this report (but we have our suspicions). So, in an effort to be truly fair and accurate, we proposed the following to Tolly:

• Veeam co-sponsors the tests and the report, and pays half the sponsorship fee;
• Veeam participates in determining the test criteria on an equal basis with Tolly and the other sponsor; or, alternatively, test criteria are determined by a mutually agreed-upon neutral third party;
• Both vendors’ representatives are present during the tests;
• The tests are conducted in September 2011. Veeam’s key people who can participate in the project are on vacation or on other projects until September 15; and
• The tests are conducted with Veeam Backup & Replication v6, which will be released in Q4 2011.

The above proposal was not accepted by Tolly, and we were told the process was too far along with the sponsoring vendor. They did tell us that we could sponsor our own report at a later time if we like.

My question now is how fair and accurate will this report be? According to the Fair Testing Charter (parts below, emphasis mine), we will be allowed to review the findings and provide feedback.

Finally, we provide them [Veeam] with the opportunity to give us an official commentary on their results, the essence of which is included in any documents we might publish.

It’s interesting they say the “essence of which.” To me it means that they will listen to our feedback but will edit it into an “essence” that puts their client in a better light. Likewise, “might publish” says to me that if Veeam were to come out ahead in the test, even with cherry-picked criteria, the report will never see the light of day.

It appears that Tolly has been doing this type of “fair and accurate” testing for quite some time; I just hope no one loses millions of dollars this time based on Tolly’s findings: http://www.brisbanetimes.com.au/business/compressed-into-nothingness-20100808-11qdh.html?from=age_ft (the Tolly report is available here)

Steve Duplessie from ESG has a great take on head-to-head testing and why ESG doesn’t do it: http://www.thebiggertruth.com/2010/03/the-politics-of-testing/

And I particularly like Simon Bramfitt’s assessment of Tolly: http://blog.simonbramfitt.com/2011/06/tolly-trumps/

Vendors in the backup space have also had their share of problems with Tolly, including Quest: http://www.infoworld.com/d/data-management/just-when-you-thought-backup-wars-were-dead-654?page=0,0[BB1]

It’s clear to us that one of our competitors intends to use this Tolly report as part of a competitive marketing campaign against us. Veeam did NOT participate in the testing and, from our vantage point, the criteria have been designed to avoid testing typical customer scenarios as well as very features that make Veeam Backup & Replication v5 stand out as the best solution on the market. So please treat any report from Tolly as suspect.

According to the report, CommVault received “Best in Class” honors for overall functionality in comparison to other solutions in the market, across seven categories encompassing technology, management, backup, restore, deduplication, media and support. CommVault outranked offerings from legacy software providers, including EMC, Symantec, CA and IBM, as well as point-level virtualization software vendors, such as Veeam.

It is important to note some facts:

1. This “Guide” is created by an analyst firm DCIG and technology advisory firm, SMB Research. But, it is distributed EXCLUSIVELY by CommVault, not DCIG.
2. CommVault also sponsors DCIG web site. DCIG has the following disclaimer on their web site: “DCIG is paid a fee by CommVault® Systems, Inc. in connection with this blog. CommVault® undertakes no obligation to update, correct or modify any statements contained in this blog; these statements represent the views and opinions of DCIG only.” See here: http://sales.dciginc.com/about/index.html. Not once does CommVault disclose this relationship with DCIG, either in the guide or in the press release mentioned above.
3. DCIG is stating that 30 vendors were identified and sent a questionnaire and 24 responses were received. The “Guide” includes Veeam as one of 24 vendors, however, Veeam never responded to the questionnaire! Here is DCIG’s description of the process: “We identified some 30 vendors who provide Virtual Server Backup Software solutions. You will find all of these vendors identified in this Buyer’s Guide. After making contact with all the vendors, we sent out a questionnaire with 130 questions. We ultimately received back 24 responses to our questionnaire.” Again, Veeam did NOT respond. How did DCIG included Veeam’s responses if Veeam didn’t respond? Is DCIG lying about the fact that they received Veeam’s response?
4. DCIG positions this document as “The Insider’s Guide” to Evaluating Virtual Server Backup Software. However the guide is NOT based on:
   1. Customer interviews, or
   2. Technical evaluations in a lab.

   Instead, the “Guide” is based on vendor responses: “After making contact with all the vendors, we sent out a questionnaire with 130 questions. We ultimately received back 24 responses to our questionnaire.” Insider means somebody who is intimately familiar. How can DCIG claim it if they purely compiled some of the vendors responses?

The EXCLUSIVE LICENSEE of the “Guide” is the vendor who is selected as the TOP vendor by the “Guide”.  If the “Guide” was paid by the vendor, can you trust it?

One of the points I liked about the Storage Switzerland article was the entire history aspect of backup vendors…

In almost every case, when a new operating system or platform became dominant, a new data protection leader emerged. Novell NetWare begat Cheyenne Software, but as UNIX and Microsoft Windows NT emerged, this NetWare solution couldn’t make the transition to the new platforms. UNIX backup drove the establishment of Legato NetWorker (eventually EMC NetWorker) and Openvision NetBackup (eventually Symantec NetBackup). Microsoft Windows NT drove the establishment of Arcadia Backup Exec (eventually Symantec Backup Exec). Even CommVault, one of the more successful newer backup applications, was driven by the establishment of Windows as an enterprise, mission-critical operating system. In short, each new operating system or platform eventually brought a new data protection application.

This truly does show that it’s a great time to be a vendor focusing on the VMware platform (and virtualization in general). Expect more great things from Veeam as we continue to innovate on these virtualization platforms.

After 1 week, I can say that the reaction has been overwhelmingly positive. Of course I know all of you can’t wait to “try it out” for yourselves. The requests for beta participation have been coming in from all over the globe and while we will have a beta program, it will rolled out in phases to ensure we get the best feedback possible from those that can devote the time to trying it out in their environments. I did give a little hint on my recent podcast with Douglas Brown over at DABCC, take a listen here.

Below are some quotes as well as links to blog posts and articles about SureBackup. If I have missed anyone I do apologize, there’s a lot of great blogs and articles out there. If you want to be included, send me an email or send me a tweet.

http://lodev.name/2010/03/22/veeam-unveils-surebackup/

The most significant feature is called Recovery Verification. With this feature, backup operators can opt to automatically verify the integrity of each and every backup.
To enable this, VB&R includes an NFS daemon that presents the images on the backup pool as a regular NFS datastore to the ESX farm. VB&R automatically registers the virtual machines (in a fenced-off network) and tests if the backup is bootable. (And doesn’t get stuck in Safe Mode, for instance.) This mechanism is OS- and application independent.

http://www.virtuallifestyle.nl/2010/03/veeam-announces-surebackup/

Cool Stuff

It makes a cool solution for actually using your DR-site for the verify process. Just replicate the backups to the second site and do the verification there. As Lode Vermeien puts it: “So I can do cross-site backup and verify at my DR site? If so, bye bye SRM.. (for 90% of my customers, who can’t afford SRM in the first place)”. It does in fact differ hugely from SRM, as SureBackup does not require array-based replication.

http://virtualisedreality.com/2010/03/22/veeam-sure-backup/

No longer do you have to hope you backups are working in-between tests as they can be verified to be working not just valid every backup, the ability to be able to granularly restore elements from any applications by starting a VM from the compressed backup file. This also introduces lots of new possibilities for testing environments etc.

http://www.van-lieshout.com/2010/03/how-sure-is-your-backup/

Universal application-item recovery

This new technology enables you to recover any virtualized application. Because of the image level backup, it’s fully OS agnostic. No special backup agents are required. It doesn’t matter which OS or even which (custom) application you are running. You can just start your VM from a backup and retrieve the items you need. Because the complete environment is started, you can use the application’s native management tools.

http://technodrone.blogspot.com/2010/03/surebackup-by-veeam.html

One of the biggest issues Administrators have with verifying the validity of Backups of an OS is that when doing this you cannot only rely on the restore only of one restored system.

http://www.vladan.fr/veeams-secret-weapon/

This applies not only to individual VMs but also to for example to VMs like Exchange 2007 which needs Active Directory and DNS to work with. All this “linked” servers must be “UP ” to be able to check your Exchange 2007 server is backed up correctly. So we might see those 3 VMs to be verified at the same time….. and declared consistent…

The new patent technology will be able to run VMs directly from compressed backup files. It’s OS and platform independent, without the need for an agent installed inside of a VM. How your actual infrastructure has to be setup?

http://www.techhead.co.uk/amazing-new-vmware-backup-product-announced-veeam-surebackup

To me this all sounds too good to be true and I am going to be interested to see how quickly it will perform these VM backup verifications, along with how many it can do concurrently and whether this is just limited by the hardware on which you are running it on. Veeam give an impressive example of the times we can expect to see: An MS Exchange server VM with 156GB of storage only takes 2 minutes to verify – wow! Now that I look forward to seeing.

http://www.virtualization.info/2010/03/veeam-announces-surebackup-technologies.html

This goes well beyond the verification of integrity for a backup image that disaster recovery vendors usually do and it’s definitively an interesting way to leverage virtualization. Veeam will support it with any guest OS (not just Windows) and any application. It will even work with Backup & Replication 4.x archives.
But the company has some news about restore too.

Less innovative, but still a very welcome addition, SureBackup also comprises the capability to restore application-level items, like emails from an Exchange Server, users from an Active Directory LDAP server or records from a SQL Server.
Proper roles and permissions for each object will be recovered too.

http://infoworld.com/d/virtualization/veeams-surebackup-transforms-vmware-image-backups-315

The process is safe, and according to Veeam, it is also fast. As an example, Veeam said it was able to bring up a backup image of a 150GB Microsoft Exchange server with 200 mailboxes in less than two minutes. The key here is that this technology takes minutes rather than hours of manually trying to do the same functionality today.

http://virtualizationreview.com/blogs/the-hoard-facts/2010/03/surebackup-coming-soon.aspx

It all sounds good, and Veeam got some pretty good analysts to provide supportive quotes, so it would seem that the only remaining challenge is getting the word out ASAP.

As Hazelman puts it, "This is going to take some time for the market to absorb because it is so different."

http://www.josemariagonzalez.es/2010/03/25/surebackup-nueva-manera-hacer-backups-vms.html

Llega SureBackup!

Creo que uno de los mejores valores de Veeam es el plantear cosas nuevas, con “pensamiento paralelo”, innovando y generando nuevas maneras de hacer las cosas. No es facil, lo hicieron con la copia sintetica decremental o inversa (siempre que la explico mis clientes asumen que es logico, pero nadie lo habia hecho antes.

Via Google Translate:

I think one of the best values of Veeam is raising new things, with "parallel thinking, innovating and creating new ways of doing things. It is not easy, it was with the synthetic copy decrement or vice versa (provided that I explain my customers assume that it is logical, but nobody had done before.

http://www.dpmworld.net/2010/03/24/veeam-surebackup-un-nuovo-modo-di-intendere-il-backup/

Recovery Verification e una soluzione pensata per verificare l’effettiva funzionalita di un backup a caldo di una macchina virtuale. I backup a caldo si appoggiano ai Volume Shadow Service di Microsoft, soluzione pensata per garantire la consistenza di un backup o di uno snapshot a caldo di una macchina virtuale. Sfortunatamente il VSS presuppone che le applicazioni siano VSS-aware: in caso contrario non viene garantita la consistenza. Tramite il Recovery Verification, Veeam vuole offrire un potente strumento in grado di avviare la macchina virtuale in un contesto separato dalla produzione per verificare non solo il corretto funzionamento del sistema operativo, ma anche (tramite appositi script) il funzionamento delle applicazioni installate.

English:

Recovery Verification is a solution designed to verify the functionality of a hot backup of a virtual machine. The hot backup will build upon the Microsoft Volume Shadow Service, a solution designed to ensure the consistency of a backup or a hot snapshot of a virtual machine. Unfortunately the VSS requires that applications are VSS-aware: otherwise there is no guarantee consistency. Through the Recovery Verification, Veeam will provide a powerful tool to start the virtual machine in an environment separate from production to test not only the proper functioning of the operating system, but also (through the provided script) the operation of installed applications.

http://www.networkworld.com/newsletters/stor/2010/032210stor2.html

Veeam verifies virtual machine recovery

Enhanced software tests recoverability of image level backups

http://www.quostarinsight.com/?p=63

The new feature included in Backup and Replication 5 answers a question which always sits at the back of IT managers minds – ‘in an emergency can I recover from my backups?’ We all have our DR drills and undertake regular test restores and rebuilds in a lab. But on the whole, if the backup product says it was successful then we trust it.

Veeam are going to stop that niggling question for machines in your virtual environments forever. Backup and Replication 5 will not only backup your machines at the high speeds seen in version 4, but now also test those backups will boot for you. Now you know they are there when you need them in a crisis.

http://vmblog.com/archive/2010/03/22/veeam-transforms-vmware-image-level-backups-with-breakthrough-technology.aspx

Recovery verification is made possible by new patent-pending technology that allows a virtual machine to run directly from a compressed backup. By publishing the content of backup files directly to ESX hosts, Veeam eliminates the need to extract backup files, and the time and storage required to do so. The technology also includes automatic creation of an isolated test environment using available production, disaster recovery (DR) or lab resources. It supports any guest operating system and virtualized application.

http://www.v3.co.uk/v3/news/2259930/veeam-introduce-verification?page=1

The problem with virtual machine backups, according to Veeam, is that companies cannot be sure whether they have a valid backup unless it is extracted and run to ensure that the application is functioning properly.

http://www.newstatesman.com/technology/2010/03/backup-veeam-test-enables

Veeam has added new a new capability to its Backup & Replication offering that will enable companies to verify the recoverability of backups from VMware virtual machines. The company says that one of the issues with current backup and restoration technology is that there is no way of verifying whether a backup has been successful or not and whether the operating system and applications will start without errors.

http://vuemuer.pmiservice.biz/?p=1144

Effettivamente si tratta di una funzione veramente utile, non vedo l’ora di mettere le mani sulle prime beta della versione 5.0 di Veeam Backup, il prodotto che integrera questa tecnologia.

Via Google Translate:

Indeed this is a really useful feature, I can not wait to get my hands on the first beta of version 5.0 of Veeam Backup, the product that will integrate this technology.

http://www.virtualizationpractice.com/blog/?p=4940

Veeam SureBackup can restore VMs on local systems, hot-site systems, or dedicated systems. VMs are restored into private virtual networks within specific resource pools so that your systems can be configured to lower the overall impact of restoration testing.

http://www.vmgu.ru/articles/veeam-surebackup-news (with some help from Google Translate)

Key components of the solution for backing up VMs on ESX server defined by the following exclusive benefits Veeam Backup:

• Самый лучший бэкап (работает быстрее всех, дедуплицируется эффективнее всех и может быть проверен на восстановление) The best backup (runs faster than anyone else, deduplitsiruetsya better all and can be checked for restoration)
• Самое лучшее восстановление (максимальная гранулярность до уровня приложений, восстановление файлов большинства ОС) The best recovery (maximum granularity to the level of applications, file recovery OS)
• Самая недорогая защита виртуальных машин с точки зрения катастрофоустойчивости (репликация от Veeam не требует больших вложений в отличие от VMware Site Recovery Manager) The most inexpensive protection of virtual machines from the standpoint of disaster-(replication of Veeam not require large investments, unlike VMware Site Recovery Manager)

http://news.techworld.com/virtualisation/3218037/veeam-claims-to-have-cracked-vm-backup-problem/?getDynamicPage

The company uses proprietary technology that, for the first time, allows a virtual machine to run direct from a compressed backup. By publishing the content of backup files direct to ESX hosts, Veeam said that it eliminated the need to extract backup files, and the time and storage required to do so. Hazelman said that the technology would work for any application that runs under VMware.

http://speicherguide.de/Magazin/StorageNews/tabid/114/articleType/ArticleView/articleId/12270/Veeam-SureBackup-veredelt-Vmware-Image-Backups.aspx

Der Backup-Prozess selbst muss nicht verandert werden

Zu der Technologie gehort auch die automatische Bereitstellung einer isolierten Testumgebung, die je nach Wunsch direkt in einer verfugbaren Produktionsumgebung oder auf Disaster-Recovery- bzw. dezidierten Testsystemen laufen kann. Es werden alle Gast-Betriebssysteme und virtualisierten Applikationen unterstutzt. »Wir wollen nicht, dass Administratoren ihre Backups verandern«, betont Hazelman. »Aber wir geben ihnen jetzt die Moglichkeit, ihre Backups zu verifizieren mit dem eindeutigen Ergebnis: Ja, dieses Backup ist gut. Und wenn eine Virtuelle Maschine (VM) recoverd werden muss, dann startet sie ganz einfach, und hat keine Fehler.«

Via Google Translate:

The backup process itself does not need to change

To the technology include the automatic provision of an isolated test environment that can run directly to each request in a production environment or on available disaster recovery or dedicated test systems. It supports all virtualized guest operating systems and applications. "We do not want that administrators" change their backups, emphasizes Hazelman. "But now we give them the opportunity to verify their backups with the clear result: Yes, this backup is good. And if a virtual machine (VM) must be recoverd, it starts easily and has no errors. "

http://blog.vmote.net/?p=135

Watching the Veeam SureBackup webcast yesterday (25/03/2010) it was quite apparent they couldn’t be more on the flip side to a typical backup vendor approach. Everything about the SureBackup technology they’ve developed is focused on the verification and testing of your backups. So much so the virtual machines don’t even require a restoration to be tested. To me this says everything I need to know that this technology, the vendor wants you to use the backed up data and not forget about it until it’s too late.

And of course my favorite Tweet from last week (and a great article too):

wcpreston New blog post: Veeam is kicking ass and taking names w/their new SureBackup product: http://www.backupcentral.com/content/view/304/47/ Thu, Mar 25 14:19:34

“Wow – you guys are smart!”

“This is unique – it's really cool. Very powerful.”

“I applaud you for this – this is a big push forward.”

“No one else is doing this!”

“This is very exciting stuff. You’re hitting on a need that’s currently not being met.”

“I wish I had had this a few years ago when I was in IT!”

“Leveraging the investment you’ve made in backups to do more”

Update: March 11, 2010 - "what you are doing is absolutely necessary and will take a lot of burden from a lot of people"

Update: March 15, 2010 - Here's a Twitter quote from Chris Wolf at Burton Group

Update: March 16, 2010: Did 4 calls back to back this morning, here's some select reactions:

"there's never been anything like this before"

"Veeam is making several Leadership moves in the new paradigm of server virtualization and enabling organizations to do new things"

"Looks so simple... how come no one ever thought of this before"

"Current process is extremely expensive and time consuming... this is the answer"

Update March 17, 2010: Just one for today...

"I'm not a guy that's easily impressed and you just impressed me"

Notice that I have not said who said the above, I won’t quote an analyst without their permission and of course they’re a bit vague too…but it’s not just us that’s excited about what we’re getting ready to release. We have many more analysts to talk to and I’m sure we’ll get more quotes, many of them may even show up in our press release with the analysts who said them.

Check back for updates!