Veeam Backup & Replication – Do you have permission for that?

This is a special guest post by Veeam SE Mike Beevor from the United Kingdom. Follow mike on Twitter @MikeBeevor. Over the last few months, I have found myself being asked more and more about Veeam’s permissions, coming face to face with the ultra-paranoid of the IT world: the Security team. Having promised to send out the Permissions Guide to a customer with a rather unusual set up, I thought that I would blog about some of the more interesting questions I’ve been asked and some of the more useful information I’ve discovered.   The customers in question are often services organizations who look after multiple large financial customers and, as such, are at the forefront of data protection, IT security and general, all-around paranoia regarding anything accessing their networks at all. The first part of the security aspect isn’t really that unusual; it involves creating the Veeam Service account, commonly specific to the Veeam product that you are using, and assigning the specific AD users or groups to that account. This generally stems from a requirement internally to work on a “needed permissions” basis, thus removing any non-critical permissions from the service account in either the vCenter or in the active directory infrastructure. An alternative to this, and we generally see this in larger environments, is to create a service account group in Active Directory specific to a particular role within the product. Backup, for example, would commonly see the following user roles with the following abilities:

Once this is done, we then look at the permissions that this role requires inside of VMware; after all, this is what we are working to back up or report upon. The list of permissions for creating backups on Veeam Backup & Replication v5.0.2 is found in the table below:

The interesting things to really note here are around the different modes that Veeam is able to back up over and subsequently, the ability to tighten the permissions required to the specific environment. Virtual Appliance mode requires the add existing disk, remove disk and change resource permissions because Veeam uses the VMware HotAdd functionality to attach the source drive to the Veeam VM to transfer the data to the target storage, and has been doing so since v3.0 of Veeam Backup & Replication. Disk lease in the SAN mode of Veeam Backup and Replication is required, as Veeam physically takes management of the VMFS volume from the ESX(i) environment for the period that it copies the data over the physical storage infrastructure. Once we’ve created the service account with the appropriate permissions, it often turns to firewalls and port numbers, so we examine the communication ports that Veeam operates across. Ports 9392, 9393 and 9394 are used for Backup and Replication Services, the backup catalog and the Veeam Enterprise Management console respectively. These ports are all standard TCP ports and can be amended to fit into a corporate firewall infrastructure. It is definitely worth checking with the security team that these ports are open and available for use without causing conflicts and having them added as trusted ports in the port database. In today’s market, VSS-enabled backups are expected, and application-aware backups are certainly at the top of the desirable list as more and more organizations need to perform object-specific recoveries into complex applications. I’ve always found DBA’s to be slightly touchy about backups, since agents can have unintended effects on the databases that they are used on, especially if they are developed in house, and additional permissions are required for the agent to be able to communicate with its relevant backup server. Veeam’s own VSS implementation for backups is innovative, using a transient piece of code to initiate a VSS flush within the application. This is done using the application’s own VSS engine with the framework provided by the operating system. The conversation soon turns to the breadth of customization of the VSS interaction on a machine basis. The long and the short of this is that a lot of secure environments will not provision an over-arching, domain-wide VSS service account, but rather rely on machine-specific VSS credentials to perform a flush. Veeam gives you the ability to set individual VSS credentials per machine within a job, ensuring that the appropriate level of permissions are maintained on a set once basis. Having done this, and dependent upon whether the application owners wish to retain the log files for their own log shipping purposes for Microsoft Exchange or Microsoft SQL, the option to truncate the logs at the start or the end of the backup, or not at all, ensures flexibility on a per machine basis. One of the longest and most protracted of the discussions, with several opinions being voiced, is where the Veeam server itself should reside in infrastructure terms, both in relation to the notion of keeping data separated and also in relation to the location of the vCenter that it was working on. This is followed swiftly by the number of Veeam servers you should have and whether you should manage them all with the Enterprise management server! To answer the point on the location of the Veeam server first, the best argument we could make is that, in the truly secure environment that we were working in, a Veeam server should be in the DMZ and should only have access to the SAN that the production VM’s were located on and the VLAN that the vCenter Server was running on. This was then expanded to include each of the Veeam servers for each segmented environment, although this presented an additional management overhead through not being able to include all Veeam servers in the Veeam management console. This does serve to reduce some of the rollup reporting capabilities for the environment as a whole, and can present some issues in the approval of Veeam virtual lab creation, which is overcome by installing the Veeam Management console on one of the Veeam servers in each segmented environment. This leads to a much needed conversation about whether SureBackup is a feature set that customers wish to implement in a secure environment. Whilst it is a no-brainer that the functionality is useful, there are some security aspects to take into consideration. Firstly, there are the additional permission requirements below, and as is evident immediately that there are a lot more “intrusive” permissions that allow a Veeam administrator to make changes to the virtualized environment itself through the Veeam console. In the case of one POC, this was considered a step too far, as the backup team had no permissions in the virtual estate, and given the choice of the virtualization team, never would! Further questions arose regarding interaction with the vCenter Server and the fact that Veeam SureBackup jobs were able to create and remove virtual machines into the virtual lab and be accessed for the potential removal of data. Whilst the second of these fears is easily allayed by the fact that the Active Directory permissions on the server in the Virtual Lab are maintained, so that the user is required to have the appropriate permissions on the application server to remove the data. Should these permissions be held however, there is the potential for data to be removed from the environment without permission.

The final piece of the POC was a puzzling one, which admittedly took me a couple of hours to resolve. Since security was key to the environment, all extraneous ports had been removed from the virtual switches inside the ESX(i) hosts to ensure that no unnecessary traffic or openings were left exploitable. This included the vmkernel port being removed from all but the vMotion VLAN, which precludes an NFS volume from being mounted on a host, and therefore prevents Instant VM Recovery being performed. So just note to have a quick check to see that the vmkernel port is present on the VeeamV if you are planning to use the feature! Wrap up and Veeam Backup & Replication v6 If you haven’t heard, we have released Veeam Backup & Replication v6, which brings in a few additional considerations; especially when vSphere 5 comes into play. In v6, at least one granular permission needs to be added. Under Roles, Virtual Machine -> Provisioning -> ‘Allow virtual machine download’ also needs to be checked to allow network mode to work. This is specific to vSphere 5 and VDDK 5.0; but is critical to provisioning security and roles for the enterprise. If you haven’t checked out v6, start with our launch page here: http://go.veeam.com/v6-backup-replication What do you do to ensure that your Veeam environment is as secure as it should be? Share your comments below!