Exchange "backed up" flag?

[kallex]

Hi!

I lost the thread where was it Gostev asked what are the reasons (if any) for not using Veeam as a sole backup tool for Exchange and alike mission critical servers.

One thing that came to mind that we bumped with few different in-machine image backuppers and with Exchange 2007 was that, the Exchange needs to know it has been backed up. Otherwise it will store some transaction logs all the way up to the level where it fills the disk.

We had a situation where Exchange disks got full and identified it as such that the Exchange clears the logs only after it recognizes the backed up state.


Let me know if you need more assistance, we can hook up with externally connected lab with Exchange set up (no ESX host access though), if you need to testdrive the backup flag.


Br,

Kalle

[Gostev]

Hello Kalle, with Veeam your Exchange transaction logs will be cleaned up after each backup (if you have Veeam VSS enabled). There are a few existing threads discussing this, here's the latest one for instance

Just please note that Exchange 2010 is not yet fully supported by Veeam VSS. Microsoft has changed a lot of things in Exchange 2010 VSS, and we are researching these changes right now.

[m.novelli]

Well, I think that Exchange and Active Directory servers require not only a full VM backup but also a backup-aware copy of Information Store and NTDS database.

I'm a long date Windows Sysadmin and I would not recommed to restore a Domain Controller or an Exchange Server just using the latest full VM copy.

Just my 2 cents

Marco

[tsightler]

Once again, Veeam fully support VSS aware snapshots of both AD and Exchange server when using the Veeam VSS Agent. Veeam doesn't just "take a VM copy", the Veeam VSS agent uses Windows VSS services to put these features into a proper, supported VSS backup state prior to taking the VM snapshot. In other words, a Veeam backup is indeed a "backup-aware copy of the Information Store and NTDS database", and it uses the Windows recommended VSS processes to achieve this.

[m.novelli]

Ok, let's assume you are right and you have full VM backup of your Domain Controllers (let's assume you have at least 2 DC in your domain)

Now imagine you have some damage to AD database (you deleted accidentally an OU or you imported a schema that created issues with your directory)

What is the procedure that you will follow to bring back AD to a consistent state?

Marco

[tsightler]

There are severals ways to do this based on whether you want to recover only the OU (non-authoritative restore of AD followed by authoritative restore leaf object), or revert your entire AD to a point before the OU (authoritative restore of entire AD). Also, are you willing to restore the entire Domain Controller or do you just want to restore AD itself. The procedures would vary based on your answers to these questions. The simplest is obviously to revert the entire domain controller/AD to the point before the error and then mark it as an authoritative restore. For that option the procedure is basically:

1. Restore the AD VM
2. Boot the VM in Directory Services Restore mode
3. Run 'ntdsutil' at the command prompt and type "authoritative restore" and then "restore database".
4. Reboot into normal mode. Since this server has been marked as authoritative for the domain it will replicate to the other domain controllers.

If you didn't want to restore the entire VM you'd need to boot the system up into Directory Service Restore mode and uses Veeam's File Level restore functionality to simply restore the NTDS and SYSVOL folders. The "System State" would not normally be needed in the Veeam restore scenario. The "SystemState" backup includes all of the components on which AD is dependent, for example system startup files, system registry, COM+ class registration database, File Replication service (the SYSVOL directory), Certificate Services database (if it is installed), Domain Name System (if it is installed), Cluster service (if it is installed). These are needed for a disaster recovery "Windows reinstall" AD restore, but with Veeam you would almost always restore the entire VM in a DR scenario, not attempt to do a Windows resinstall and then a "pick and choose" file level restore. The restore of the VM would restore the system state as well.

I really don't think restoring AD is that different when using other tools. Our previous backup tool used "SystemState" and and "Active Directory" backup API's but a restore still required booting the system into Directory Services Restore mode, restoring the "AD backup" and running ntdsutil commands to complete the restore. There might have been some check boxes that would run these command for you (it seems like there was a "Perform an Authoritative Restore" checkbox) but overall the procedure was about the same.

Please note that this is note that the above is not an attempt to provide a complete guide to AD recovery with Veeam, only to answer the question about "What is the procedure". The actual steps we would take would vary based on the nature of the issue we experienced, it's ability to be corrected without a restore (we consider the restore of AD to be a last-resort option). The point really is, all you truly need to restore AD is a consistent backup of the AD components, and Veeam, using VSS provides that.

[m.novelli]

Long story short: this procedure doesn't work for a restored Domain Controller from an image-level backup (Veeam Backup, VCB-integrated backup, SAN snapshot)

The DC will start with the netlogon service paused and in Event Viewer you will find the error "The Active Directory database has been restored using an unsupported restoration procedure"

Then you will not be able to autorithative restore the directory objects.

I've personally tested this procedure with Windows 2000 and Windows 2003 DC. Not yet on Windows 2008 DC.

Let's look at this KB: http://support.microsoft.com/kb/875495

This is the actual Microsoft recommendation about backupping AD related data: http://support.microsoft.com/kb/888794

"To roll back the contents of Active Directory to a previous point in time, restore a valid system state backup. A system state backup can be restored up to the tombstone lifetime number of days after the backup was performed. The backup must have also been made on the same operating system installation as the operating system that you are restoring.

Active Directory does not support other methods to roll back the contents of Active Directory. In particular, Active Directory does not support any method that restores a snapshot of the operating system or the volume the operating system resides on. This kind of method causes an update sequence number (USN) rollback. When a USN rollback occurs, the replication partners of the incorrectly restored domain controller may have inconsistent objects in their Active Directory databases. In this situation, you cannot make these objects consistent."

I'm feeling the same recommendation exist for Exchange CCR and mirrorer SQL Servers

Marco

[Gostev]

Thank you very much Tom for taking your time to write this guide. I am on vacation until Jan 10th, which explains my very sporadic forum attendance lately.

Long story short: this procedure doesn't work for a restored Domain Controller from an image-level backup (Veeam Backup, VCB-integrated backup, SAN snapshot)

Marco, you are 100% correct, just restoring latest VM copy of DC for example will cause USN rollback which completely trashes your DC (you have to demote it, and go through pains of manually cleaning up references to old DC in AD configuration).

This is exactly why Veeam is shipping proprietary VSS integration module for "proper" backups and restores. Our agent executes automatically before the actual VM snapshot is created, if you have Veeam VSS enabled, of course. Just investigate the Windows Event log for DC/Exchange/SQL/etc. after backup with Veeam, and you will see the corresponding VSS events there. Also, after restore with Veeam, you will see events of successful shadow copy restore.

I actually have 1.5 years old videos for both scenarios (restoring simple DC VM copy, and restoring Veeam DC backup).
Preparing the test lab (2 DCs) (Windows 2003)
Restoring regular DC VM copy (ouch, USN rollback)
Restoring Veeam DC backup (feel the power of Veeam VSS)

While you are mentioning DC and Exchange, it should be noted that these two require even more complex VSS backup/restore approach than other VSS-aware apps, as Microsoft requires certain custom restore procedures performed for these applications to ensure successful restore. For example, DC should be first booted into the safe mode (Directory Services Restore mode) to ensure Active Directory files are not locked by additional processes like antivirus when VSS restore is being performed). This is something Veeam VSS also implements, and it is fully automated - as you can see from the video above, no manual steps are required (well, in case of non-authoritative restore, and Tom has already covered the authoritative one).

All this functionality is actually unique to Veeam among all image-level backups... yet another reason to choose Veeam - by the way, we have this functionality since 2008.

[Gostev]

I've personally tested this procedure with Windows 2000

By the way, please keep in mind that Windows 2000 does not have VSS at all.

[tsightler]

I've personally tested this procedure with Windows 2000 and Windows 2003 DC. Not yet on Windows 2008 DC.

So you personally tested this with the Veeam VSS agent enabled and working? That's not really possible with Windows 2000 since it doesn't have VSS. With Windows 2003 it does work, I've tested it without issue.

Let's look at this KB: http://support.microsoft.com/kb/875495

This is the actual Microsoft recommendation about backupping AD related data: http://support.microsoft.com/kb/888794

I'm quite aware of the requirements for backup of AD and the Microsoft articles are correct, simply taking a snapshot and rolling it back is NOT a valid way to backup AD. Fortunately, that's also NOT what Veeam does. Assuming you enable it, Veeam uses VSS to place AD into a consistent state prior to taking the snapshot, the snapshot is then made, and then VSS is signaled to return to normal operations. When you preform a restore of the VM you boot the system into Directory Services Restore mode and, since VSS had the NTDS database in a consistent state, you CAN perform an authoritative restore.

Since you like to link to MS articles here's a like on VSS Backup and Restore of AD:
http://msdn.microsoft.com/en-us/library ... 85%29.aspx

Yes, Exchange and SQL have the same requirements, but assuming you have the VSS writers for these applications installed and they are working properly, and you use Veeam VSS agent, then yes they will be backed up in a consistent state as well. Notice that Exchange even purges the logs. Veeam doesn't do that itself, it signals the VSS writer that the "backup" was complete (actually just a snapshot) and the VSS writer purges the logs when it "unfreezes".

[donikatz]

Obviously neither Tom nor Anton need my help here, but maybe some real-world testimony would make you feel more comfortable? Not only have I tested this, I've performed a *production* restore of a w2k3 DC with Veeam and it worked exactly and as simply as in the video. I've also done several *production* SQL restores without issue. Veeam also works well in our Exchange restore tests, although we haven't had to do any in production (knock on wood). Although agent-based apps like Backup Exec may have more direct hooks for simpler granular restore (we still use BE for Exchange brick-level restores because our admins are more familiar with the process), Veeam is more than capable without the drawbacks of an agent. I hope to move away from BE altogether for Exchange this year; it's just a matter of updating our runbook and training. Honestly, if there's one area you certainly don't need to lose sleep over with Veeam, it's with Microsoft products. MS has well-proven APIs and Veeam makes great use of them; Veeam VSS is excellent. Heck, if only Oracle on Linux had VSS the way it does on Windows it would make my life a lot easier...

[m.novelli]

Well guys, you are right
My direct restore experience was with a Windows 2000 DC (not supporting VSS) and with a Windows 2003 DC that now I suppose wasn't backupped with VSS integration

Marco

[tsightler]

BTW, I do want to say that I would never criticize the idea of using a secondary backup method for critical information. It never hurts to have more than one way to restore a system. For example, we use Oracle RMAN to backup many of our Oracle databases, even though most of them are already backed up with Veeam. This way the DBA's can preform their own restores using the technology that their familiar with, but we still have Veeam backups for DR restores of the entire system. We've used both methods for restores of production systems with great success.

[donikatz]

we use Oracle RMAN to backup many of our Oracle databases, even though most of them are already backed up with Veeam. This way the DBA's can preform their own restores using the technology that their familiar with, but we still have Veeam backups for DR restores of the entire system.

Tom-- Not to go too off-topic, but do you use an Oracle freeze/thaw script for the Veeam backups or do you rely on RMAN to recover if Oracle doesn't come back up clean? Also, what Oracle version do you use? We're in the process of optimizing our Oracle infrastructure (including P2Ving the last of our physical clusters), so very interested in your experience. Thanks!

[Gostev]

Doni, if you are running Oracle on Windows, Oracle VSS is a good option - see here: How to backup Oracle. Also in this topic there are some commands mentioned which you can use instead of VSS with the pre-freeze/post-thaw scripts (for example, on Linux).