The Business Challenge
On December 24th, 2019, alarm bells started sounding to warn that the
RACE had been the victim of a ransomware attack, and its servers, data
areas, virtual machines (VM’s), user devices and its entire infrastructure was
being encrypted. This, according to Amaro Morales, Head of Architecture
and Technology at the company, was one of the most intense situations
he has faced in his professional career.
“In terms of risk mapping, the most critical problem lies in not being able
to guarantee support services, which could lead to the loss of a strategic
account, such as an insurance company. In a sector such as this one, the
average contract can be worth between €10-15 million. A loss like this can
have a domino effect, meaning that the following year up to 30% of accounts
could be lost,” commented Morales. This, without a doubt, highlights why
it was of vital importance for the RACE to recover from this emergency.
What makes the effect that this attack had even worse was that the
company’s previous backup platform stopped operating, which meant
that they had to act quickly and effectively in order to immediately create
a parallel backup solution for the entire infrastructure to ensure that its
services came back online while guaranteeing that their operations would
not be affected again.
The RACE’s infrastructure is composed of 90% legacy systems hosted
on operational CPD’s on four production virtualization servers that host
around 140 VM’s. In total, The IT area stores approximately 32 terabytes
of data. Morales contacted the Velorcios Group, a Veeam Cloud & Service
Provider (VCSP), in order to delegate backup services, leveraging their
infrastructure and cloud environment.
In total, the attack halted the business’ operations for less than 8 hours.
Thanks to their hard work, Morales and his team of ten, which offers services
to around 850 employees, successfully recovered close to 95% of the
Velorcios leveraged the fact that the company is a member of the VCSP
program, which offers it the opportunity to use Veeam’s solution ecosystem
and technology to provide the RACE with a reliable and high-quality
service. Meanwhile, the RACE’s IT department benefits from the managed
service model offered by Velorcios, as it no longer needs to focus on backup
management and can instead turn its attention to supporting the
business and driving innovation, safe in the knowledge that all its data and
infrastructure are protected.
Thanks to the fact that the RACE took advantage of this model, almost
immediately migrating to Velorcios’ infrastructure and leveraging its
experience with the Veeam platform, in total, this attack only affected the
services offered by the organization for 8 hours.
With a lot of hard work, Amaro Morales and his team (comprising 10 members
to assist a workforce of around 850 employees) successfully recovered 95%
of all the comprised data, in addition to the rest of the servers and services
that had been affected. For those environments that required a different
strategy, the servers were mounted one by one from scratch. The Veeam
solution, provided by Velorcios, enabled them to guarantee that everything
that was being mounted had a reliable backup point in order to recover and
restore information, in addition to recovery points.
The Veeam Solution
In December 2019, over the holidays, the RACE found itself facing the need
to implement the Velorcios service model, and it began working with Veeam
technology with no testing period whatsoever. It was a great surprise for the
IT department to discover Veeam’s level of integration, not to mention the
fact that it could be implemented over the existing back-up platform with
no compatibility problems or the need for additional equipment.
Guillermo Álvarez, Infrastructure Manager at the RACE, highlights the speed,
ease and transparency of the entire process, which resulted in a tool that
really helped the company overcome a major threat and get back up and
running in a short period of time.
“In terms of compliance and internal and external audit management,
we have easily integrated reporting alarms to our Support Solutions, and
it is Veeam itself that creates the service tickets and our environments,
ensuring transparency and facilitating the work that we do,” commented the
Head of Architecture and Technology.
The RACE is now more secure and complete in terms of back-up and
recovery tasks. All its VM’s in both production and development are
protected; data deduplication and compression features enabled several
recovery states; it is possible to make traditional on-site backups, using
an LTO robot; and everything is now easier than before thanks to the
versatility and capabilities of Veeam.
According to Amaro Morales, when working with the
organization’s previous provider operations were disrupted. “The
machine normally froze when trying to perform a back-up, which
meant that we ended up with a dirty copy,” commented Amaro
Morales. “Now, I can have up to 18 states and the VM isn’t even
affected; in a matter of minutes, I can perform a rollback or create
a clone of the machine for troubleshooting, analysis, etc. This means
that we can make full use of our time as we no longer have to assign
someone to work on this for 3 days”.
In fact, thanks to Veeam, the IT department no longer needed
a technician to focus solely on back-ups because its trusted
partner, Velorcios, now covers this.
“The number of snapshots and Veeam options that can be used
to recover data, be it a VM or a file, has made all the difference,”
commented Álvarez. In terms of the business, this has allowed them
to greatly improve response times: in the past someone had to wait
for 2 or 3 days for the IT department to recover data from a specific
folder, while today this can be done in 4 hours (or even sooner
if dealing with a high priority ticket).
For Morales, all of this has increased employee satisfaction, which
is priceless and more important than any type of saving. It is important
to highlight the fact that, for OPEX costs similar to those of its
previous back-up tool, the company now has access to a range
of benefits in terms of both services and functionality thanks
to Veeam. And the future holds even better news for the company.
In conjunction with Velorcios, the IT department at the RACE is about
to launching a proyect to take full advantage of the Replication
Management part of Veeam Backup to help offer high availability for
its critical VM’s and servers, mounting a decentralized DR at Velorcios’
facilities in the Canary Islands.
According to Alberto San Millán, Territory Manager for Velorcios
for Peninsular Spain, they are also working with the RACE to exploit
granularity for database recovery (a file-level approach is currently
being used) and comply with regulations focusing on the elimination
of database registries.
“We are showing the RACE that Veeam is a truly comprehensive
solution, and when they come to us with a new requirement, we can
resolve it using the same tool,” added San Millán.
Amaro Morales also talked about his excitement at learning more about
what Veeam has to offer in order to continue streamlining IT services
at the RACE.
- Quick and efficient deployment to tackle the threat, save millions
of Euros and ensure added value.
Thanks to the support offered by Veeam, in less than 8 hours the RACE
began to recover from the emergency caused by the ransomware attack,
which could have cost millions of Euros (for the loose of strategic
accounts). Furthermore, it drastically improved efficiency, functionality
and response times compared to the company’s previous backup
solution (recovery times now stand at 4 hours, compared to 2 or 3 days),
and Veeam is also having a positive impact on compliance and audit
- Ease of use and transparent technological integration
“We had to implement Veeam straight away and from scratch, without
really knowing anything about the solution,” commented Guillermo
Álvarez, Infrastructure Manager at the RACE, who values and is thankful
that this solution has been implemented so quickly and easily. They also
reused the architecture they had in place without having to add anything
else, and Veeam was transparently integrated into the company’s entire
- Reliable and agile back-up and recovery to ensure happy clients.
“There are 10 people on the team, and we need to be extremely focused,”
commented Amaro Morales, Head of Architecture and Infrastructure at the
RACE. “Thanks to Veeam, ticket management processes are no longer
a headache, and I was also able to reassign a post as we no longer needed
a technician to focus solely on backup management”.