Thanks to Veeam, the RACE saves millions of Euros by recovering from ransomware, and optimizes its IT services

The Business Challenge

On December 24th, 2019, alarm bells started sounding to warn that the RACE had been the victim of a ransomware attack, and its servers, data areas, virtual machines (VM’s), user devices and its entire infrastructure was being encrypted. This, according to Amaro Morales, Head of Architecture and Technology at the company, was one of the most intense situations he has faced in his professional career.

“In terms of risk mapping, the most critical problem lies in not being able to guarantee support services, which could lead to the loss of a strategic account, such as an insurance company. In a sector such as this one, the average contract can be worth between €10-15 million. A loss like this can have a domino effect, meaning that the following year up to 30% of accounts could be lost,” commented Morales. This, without a doubt, highlights why it was of vital importance for the RACE to recover from this emergency.

What makes the effect that this attack had even worse was that the company’s previous backup platform stopped operating, which meant that they had to act quickly and effectively in order to immediately create a parallel backup solution for the entire infrastructure to ensure that its services came back online while guaranteeing that their operations would not be affected again.

The RACE’s infrastructure is composed of 90% legacy systems hosted on operational CPD’s on four production virtualization servers that host around 140 VM’s. In total, The IT area stores approximately 32 terabytes of data. Morales contacted the Velorcios Group, a Veeam Cloud & Service Provider (VCSP), in order to delegate backup services, leveraging their infrastructure and cloud environment.

In total, the attack halted the business’ operations for less than 8 hours. Thanks to their hard work, Morales and his team of ten, which offers services to around 850 employees, successfully recovered close to 95% of the compromised data.

Velorcios leveraged the fact that the company is a member of the VCSP program, which offers it the opportunity to use Veeam’s solution ecosystem and technology to provide the RACE with a reliable and high-quality service. Meanwhile, the RACE’s IT department benefits from the managed service model offered by Velorcios, as it no longer needs to focus on backup management and can instead turn its attention to supporting the business and driving innovation, safe in the knowledge that all its data and infrastructure are protected.

Thanks to the fact that the RACE took advantage of this model, almost immediately migrating to Velorcios’ infrastructure and leveraging its experience with the Veeam platform, in total, this attack only affected the services offered by the organization for 8 hours.

With a lot of hard work, Amaro Morales and his team (comprising 10 members to assist a workforce of around 850 employees) successfully recovered 95% of all the comprised data, in addition to the rest of the servers and services that had been affected. For those environments that required a different strategy, the servers were mounted one by one from scratch. The Veeam solution, provided by Velorcios, enabled them to guarantee that everything that was being mounted had a reliable backup point in order to recover and restore information, in addition to recovery points.

The Veeam Solution

In December 2019, over the holidays, the RACE found itself facing the need to implement the Velorcios service model, and it began working with Veeam technology with no testing period whatsoever. It was a great surprise for the IT department to discover Veeam’s level of integration, not to mention the fact that it could be implemented over the existing back-up platform with no compatibility problems or the need for additional equipment.

Guillermo Álvarez, Infrastructure Manager at the RACE, highlights the speed, ease and transparency of the entire process, which resulted in a tool that really helped the company overcome a major threat and get back up and running in a short period of time.

“In terms of compliance and internal and external audit management, we have easily integrated reporting alarms to our Support Solutions, and it is Veeam itself that creates the service tickets and our environments, ensuring transparency and facilitating the work that we do,” commented the Head of Architecture and Technology.

The RACE is now more secure and complete in terms of back-up and recovery tasks. All its VM’s in both production and development are protected; data deduplication and compression features enabled several recovery states; it is possible to make traditional on-site backups, using an LTO robot; and everything is now easier than before thanks to the versatility and capabilities of Veeam.

According to Amaro Morales, when working with the organization’s previous provider operations were disrupted. “The machine normally froze when trying to perform a back-up, which meant that we ended up with a dirty copy,” commented Amaro Morales. “Now, I can have up to 18 states and the VM isn’t even affected; in a matter of minutes, I can perform a rollback or create a clone of the machine for troubleshooting, analysis, etc. This means that we can make full use of our time as we no longer have to assign someone to work on this for 3 days”.

In fact, thanks to Veeam, the IT department no longer needed a technician to focus solely on back-ups because its trusted partner, Velorcios, now covers this.

“The number of snapshots and Veeam options that can be used to recover data, be it a VM or a file, has made all the difference,” commented Álvarez. In terms of the business, this has allowed them to greatly improve response times: in the past someone had to wait for 2 or 3 days for the IT department to recover data from a specific folder, while today this can be done in 4 hours (or even sooner if dealing with a high priority ticket).

For Morales, all of this has increased employee satisfaction, which is priceless and more important than any type of saving. It is important to highlight the fact that, for OPEX costs similar to those of its previous back-up tool, the company now has access to a range of benefits in terms of both services and functionality thanks to Veeam. And the future holds even better news for the company.

In conjunction with Velorcios, the IT department at the RACE is about to launching a proyect to take full advantage of the Replication Management part of Veeam Backup to help offer high availability for its critical VM’s and servers, mounting a decentralized DR at Velorcios’ facilities in the Canary Islands.

According to Alberto San Millán, Territory Manager for Velorcios for Peninsular Spain, they are also working with the RACE to exploit granularity for database recovery (a file-level approach is currently being used) and comply with regulations focusing on the elimination of database registries.

“We are showing the RACE that Veeam is a truly comprehensive solution, and when they come to us with a new requirement, we can resolve it using the same tool,” added San Millán.

Amaro Morales also talked about his excitement at learning more about what Veeam has to offer in order to continue streamlining IT services at the RACE.

The Results

• Quick and efficient deployment to tackle the threat, save millions of Euros and ensure added value.
  Thanks to the support offered by Veeam, in less than 8 hours the RACE began to recover from the emergency caused by the ransomware attack, which could have cost millions of Euros (for the loose of strategic accounts). Furthermore, it drastically improved efficiency, functionality and response times compared to the company’s previous backup solution (recovery times now stand at 4 hours, compared to 2 or 3 days), and Veeam is also having a positive impact on compliance and audit management tasks.
• Ease of use and transparent technological integration
  “We had to implement Veeam straight away and from scratch, without really knowing anything about the solution,” commented Guillermo Álvarez, Infrastructure Manager at the RACE, who values and is thankful that this solution has been implemented so quickly and easily. They also reused the architecture they had in place without having to add anything else, and Veeam was transparently integrated into the company’s entire technology environment.
• Reliable and agile back-up and recovery to ensure happy clients.
  “There are 10 people on the team, and we need to be extremely focused,” commented Amaro Morales, Head of Architecture and Infrastructure at the RACE. “Thanks to Veeam, ticket management processes are no longer a headache, and I was also able to reassign a post as we no longer needed a technician to focus solely on backup management”.