On December 24th, 2019, alarm bells started sounding to warn that the RACE had been the victim of a ransomware attack, and its servers, data areas, virtual machines (VM’s), user devices and its entire infrastructure was being encrypted. This, according to Amaro Morales, Head of Architecture and Technology at the company, was one of the most intense situations he has faced in his professional career.
“In terms of risk mapping, the most critical problem lies in not being able to guarantee support services, which could lead to the loss of a strategic account, such as an insurance company. In a sector such as this one, the average contract can be worth between €10-15 million. A loss like this can have a domino effect, meaning that the following year up to 30% of accounts could be lost,” commented Morales. This, without a doubt, highlights why it was of vital importance for the RACE to recover from this emergency.
What makes the effect that this attack had even worse was that the company’s previous backup platform stopped operating, which meant that they had to act quickly and effectively in order to immediately create a parallel backup solution for the entire infrastructure to ensure that its services came back online while guaranteeing that their operations would not be affected again.
The RACE’s infrastructure is composed of 90% legacy systems hosted on operational CPD’s on four production virtualization servers that host around 140 VM’s. In total, The IT area stores approximately 32 terabytes of data. Morales contacted the Velorcios Group, a Veeam Cloud & Service Provider (VCSP), in order to delegate backup services, leveraging their infrastructure and cloud environment.
In total, the attack halted the business’ operations for less than 8 hours. Thanks to their hard work, Morales and his team of ten, which offers services to around 850 employees, successfully recovered close to 95% of the compromised data.
Velorcios leveraged the fact that the company is a member of the VCSP program, which offers it the opportunity to use Veeam’s solution ecosystem and technology to provide the RACE with a reliable and high-quality service. Meanwhile, the RACE’s IT department benefits from the managed service model offered by Velorcios, as it no longer needs to focus on backup management and can instead turn its attention to supporting the business and driving innovation, safe in the knowledge that all its data and infrastructure are protected.
Thanks to the fact that the RACE took advantage of this model, almost immediately migrating to Velorcios’ infrastructure and leveraging its experience with the Veeam platform, in total, this attack only affected the services offered by the organization for 8 hours.
With a lot of hard work, Amaro Morales and his team (comprising 10 members to assist a workforce of around 850 employees) successfully recovered 95% of all the comprised data, in addition to the rest of the servers and services that had been affected. For those environments that required a different strategy, the servers were mounted one by one from scratch. The Veeam solution, provided by Velorcios, enabled them to guarantee that everything that was being mounted had a reliable backup point in order to recover and restore information, in addition to recovery points.
In December 2019, over the holidays, the RACE found itself facing the need to implement the Velorcios service model, and it began working with Veeam technology with no testing period whatsoever. It was a great surprise for the IT department to discover Veeam’s level of integration, not to mention the fact that it could be implemented over the existing back-up platform with no compatibility problems or the need for additional equipment.
Guillermo Álvarez, Infrastructure Manager at the RACE, highlights the speed, ease and transparency of the entire process, which resulted in a tool that really helped the company overcome a major threat and get back up and running in a short period of time.
“In terms of compliance and internal and external audit management, we have easily integrated reporting alarms to our Support Solutions, and it is Veeam itself that creates the service tickets and our environments, ensuring transparency and facilitating the work that we do,” commented the Head of Architecture and Technology.
The RACE is now more secure and complete in terms of back-up and recovery tasks. All its VM’s in both production and development are protected; data deduplication and compression features enabled several recovery states; it is possible to make traditional on-site backups, using an LTO robot; and everything is now easier than before thanks to the versatility and capabilities of Veeam.
According to Amaro Morales, when working with the organization’s previous provider operations were disrupted. “The machine normally froze when trying to perform a back-up, which meant that we ended up with a dirty copy,” commented Amaro Morales. “Now, I can have up to 18 states and the VM isn’t even affected; in a matter of minutes, I can perform a rollback or create a clone of the machine for troubleshooting, analysis, etc. This means that we can make full use of our time as we no longer have to assign someone to work on this for 3 days”.
In fact, thanks to Veeam, the IT department no longer needed a technician to focus solely on back-ups because its trusted partner, Velorcios, now covers this.
“The number of snapshots and Veeam options that can be used to recover data, be it a VM or a file, has made all the difference,” commented Álvarez. In terms of the business, this has allowed them to greatly improve response times: in the past someone had to wait for 2 or 3 days for the IT department to recover data from a specific folder, while today this can be done in 4 hours (or even sooner if dealing with a high priority ticket).
For Morales, all of this has increased employee satisfaction, which is priceless and more important than any type of saving. It is important to highlight the fact that, for OPEX costs similar to those of its previous back-up tool, the company now has access to a range of benefits in terms of both services and functionality thanks to Veeam. And the future holds even better news for the company.
In conjunction with Velorcios, the IT department at the RACE is about to launching a proyect to take full advantage of the Replication Management part of Veeam Backup to help offer high availability for its critical VM’s and servers, mounting a decentralized DR at Velorcios’ facilities in the Canary Islands.
According to Alberto San Millán, Territory Manager for Velorcios for Peninsular Spain, they are also working with the RACE to exploit granularity for database recovery (a file-level approach is currently being used) and comply with regulations focusing on the elimination of database registries.
“We are showing the RACE that Veeam is a truly comprehensive solution, and when they come to us with a new requirement, we can resolve it using the same tool,” added San Millán.
Amaro Morales also talked about his excitement at learning more about what Veeam has to offer in order to continue streamlining IT services at the RACE.
The Royal Automobile Club of Spain (RACE) was founded in 1903 to promote motor racing and the use of automobiles, driving collaboration in order to search for solutions plaguing the drivers who spearheaded the motoring movement in Spain. The RACE’s vision is to become a global motoring benchmark in Spain, in its dual role as a “Club for Motoring Enthusiasts” and an “Expert Institution” for the authorities, business associations, clients and consumers.
Their mission as the leading motoring club in Spain is to spearhead innovation, offering its partners and client best-in-class roadside assistance and a series of unique services within the motoring world and travel experiences. The RACE represents and defends motorists before national and international institutions and organizations thanks to its experience and knowhow in areas such as motorsports and traffic regulations.
Over the past 15 years, its business model has evolved to become a car club, offering a range of services relating to motoring and entertainment. It provides services to 8 million customers in Spain, and it is part of a European holding that guarantees roadside assistance to all its members, domestically and internationally.
The RACE has 5 companies and is a founding member of the International Automobile Federation (FIA). Some of the services it offers include its racetrack for motorsport events and other related activities, a sports club, and a cultural foundation that has one of the largest databases of technical manuals, magazines, articles and maps, in addition to being a benchmark in the motoring world.
The RACE’s Architecture and Technology department is a strategic ally whose mission is to guarantee the operation of every system component, as well as the aggregation of other insurance companies and the integration of services between support platforms, offering a network of employees and support that encompasses the country as a whole.
The major challenge facing the RACE’s IT department lies in maintaining the availability of operations and the services that the company offers its clients 24/7, 365 days a year, complying with the highest and most demanding levels of quality across the company’s 5 critical areas. Any loss of service or low levels of service is critical for the company as it would fail in delivering its guaranteed minimum service levels.
Not being 100% available and operational could mean that the RACE is not offering its partners the proper support in the event of an incident while travelling, not being able cover them when an accident occurs, or not being able to accompany them in the event of any unforeseen circumstance or requirement.
The important collaboration and service agreements the RACE has with insurance companies, car manufacturers and other organizations come with models to assess service provision, calibrate development and measure service quality. Compliance issues in these agreements, no matter their nature, could lead to significant sanctions or even a contract not being renewed.
This represented a major risk when a ransomware hit.