Automate recovery verification of VM backups with Veeam SureBackup

In this special guest post, Andrey Konovalov provides a deep-dive of his strategy with Veeam’s SureBackup functionality.

image

As I was exploring possibilities of Veeam Backup & Replication, I took a close look at SureBackup – the new functionality that lets you automatically verify the recoverability of virtual machine (VM) backups.

So, what does SureBackup actually do?

With Veeam Backup & Replication, you can publish and start a VM right from a backup in a Virtual Lab. A virtual lab is a dedicated environment isolated from the rest of your infrastructure. Every virtual lab includes a set of isolated networks, virtual switches and port groups. It is important to note that these objects do not connect to any physical networks. This avoids any contention with production virtual machines [rv1] . [hk2] When you create a virtual lab, you select one or more ESX(I) hosts on which your VMs will run, choose storage that will keep temporary log files, and define network settings.

The other key component to a virtual lab is the Application Group. The Application Group contains VMs you want to verify. In the simplest case, the group can have just one VM; in most cases, however, there are a number of VMs to verify. For example, to check application servers—a web server or a mail server— you may need to deploy infrastructure services (such as DNS, directory service and database servers). When you create the group, you should define in which order the VMs will be started, and what tests will be used to verify each of these VMs.

By default, Veeam Backup & Replication offers a number of predefined test sets also known as Server Roles. When you create an application group, you can assign each VM a specific role to define what tests will be performed for its verification. Role definitions are stored in .xml files in the %ProgramFiles%VeeamBackup and ReplicationSbRoles folder. You can modify these files to match your requirements. There are several ways to verify your VM using SureBackup:

  • Get a heartbeat signal from the VMware Tools agent running inside a VM
  • Send a ping request to a VM network address
  • Execute test scripts to check whether the applications inside VM are working correctly

Testing the VM heartbeat is obvious enough; but the other two options allow a more granular verification configuration.

SureBackup Isolated Network Configuration

To access an isolated network, Veeam uses a specific virtual machine— a proxy server appliance that is created automatically when you set up your virtual lab. This proxy server uses one of its interfaces to connect to a production server and another one to connect to an isolated network. If there are several subnets in your production infrastructure, the proxy server will also act as a router between these isolated networks. As a rule, the proxy server appliance should acquire router’s IP address in each of the subnets on the isolated networks side.

image

Both production and isolated networks use the same range of IP addresses. To avoid IP address conflicts when accessing VM in isolated network from production network, Veeam Backup & Replication uses network masquerading. Every IP address of a node in an isolated network acquires a similar IP address in the masquerade network: for example, 192.168.1.0/24 goes to 192.168.254.0/24. When you start a virtual lab, a static route to a masquerade network is added automatically on the Veeam Backup & Replication server. The proxy server appliance’s production network IP address is used as a gateway address.

When the Veeam server needs to access a verified VM, it uses the VM’s masquerade network address rather than the production address (for example, 192.168.1.36 goes to 192.168.254.36). According to the routing table, the packet is sent to the proxy server.

image

The proxy server here functions as a NAT device. It receives the packet and changes the sender’s IP address to its own IP address in the isolated network (in our example, 192.168.1.1). It also changes the masquerade network IP address to the production network IP address, and then sends the packet to a receiver in the isolated network. This can be monitored with a tool such as Microsoft Network Monitor.

Therefore, while the production and test networks are completely separated, the Veeam Backup server still has the ability to communicate with verified VMs.

image

SureBackup Script Creation and Management

Finally, with Veeam Backup & Replication, you can run scripts to verify VMs. All predefined roles use the same script – VmConnectionTester.exe, a simplified counterpart of a Telnet. Arguments of the script are an IP address of the server and a TCP port that need to be verified. A small but important comment here— Veeam Backup & Replication considers that the test has passed successfully if the script is completed with the return code 0. If it is completed with any other code, the test has failed.

To verify VMs, you can use your own custom scripts—Veeam Backup & Replication supports .exe, .bat, .cmd, .js, .vbs, .wsf files. To run scripts, you need to be logged in under the same account that is used to run Veeam Backup & Replication services.

In general, with VmConnectionTester.exe you can verify whether network services are available on the VM. However, there are services that do not use the TCP protocol—these are client services and services running locally. To verify these services, you can use the following PowerShell script (query-service.ps1) which checks if the service on a remote machine is started:

#query-service.ps1
param(
[string] $ip, #IP address of checked server
[string] $service ) #Service name
$result = (get-Service -ComputerName $ip -Name $service -ErrorAction SilentlyContinue)
if($result.status -eq "Running")
{
exit
}
else
{
write-host ("Error 1, Service '" + $service + "' not running or not found.") #if service not found or not running, then echo
$host.SetShouldExit(1)
exit
}

The script takes an IP address of the VM and a name of the service as input parameters. If the service is not started or there is no such service, the script will terminate and pass error code 1 to the Veeam Backup handler. For the script to run as a PowerShell script (.ps1) on the Veeam Backup & Replication server, the execution policy must be set to ” remotesigned”. To do this, run the following command in the PowerShell console:

set-executionpolicy remotesigned

Next, you need to make sure that the account under which the script will run has the required permissions on the remote machine (for example, this account can be a part of the local Administrators group). This is required to get information about started services.

And finally, the Windows firewall on the remote server should allow remote connections. To configure it, you can enable corresponding rules or turn off the firewall.

image

Because Veeam Backup & Replication does not allow you to run PowerShell script files (.ps1) directly, you can use a batch file as a shell (query-service.bat):

REM query-service.bat
@ECHO OFF
powershell.exe -noninteractive -noprofile -command "& {C:Tempquery-service.ps1 %1 %2 }"
EXIT /B %errorlevel%

Now you need to save both files to any folder on the server (by default, I used C:Temp) and edit one of the application groups by adding a script, define a path to the query-service.bat file in the Path field, specify the VM IP address (in the %vm_ip% variable) and the service name as arguments. In this example I verified a DHCP client.

image

Now, the VM will pass the test if the script completes successfully and the service is started. If the service is not started for some reason, Veeam Backup will notify you about it and automatically power off the virtual lab.

image

SureBackup – A Wealth of Options

In this post, the deep dive describes one of the options that help automate recovery verification of VM backups — a routine and rather boring task that many backup administrators have to perform. If you do not normally perform backup verifications, this article may give you a good reason to start doing so. There is actually nothing more useless than a backup that cannot be recovered.

Andrey Konovalov is a Senior Engineer at Helios IT Solutions and writes the VM Press blog at http://vmpress.blogspot.com/ and is on Twitter @omnimod.

Stay up to date on the latest tips and news
By subscribing, you are agreeing to have your personal information managed in accordance with the terms of Veeam’s Privacy Policy
You're all set!
Watch your inbox for our weekly blog updates.
OK