There are two words that strike terror in the heart of any IT professional: system crash. If the next two words are data loss, the situation escalates from bad to worse.
That’s what happened when an IT system containing private identifiable information (PII) including employees’ names, addresses and social security numbers crashed at a federal government agency.
“We were backing up and replicating our PII system three times each day to four locations on-premises,” explained the IT Project Manager who experienced the situation firsthand. “We thought there was no way this system could ever crash, but we were dead wrong.”
The first crash occurred two days before Thanksgiving. Several members of the IT team were taking time off, so they immediately returned to the office and spent the next 36 hours trying to resolve the problem. Even though they had hardware redundant storage for the virtual server platform, the aged storage just crashed without warning; it corrupted most of the VM data.
“We tried restoring the 30 virtual machines (VMs) supporting the PII system from backups, but our backup software (Syncsort) failed us miserably,” he said. “At one point we feared the system was unfixable and data was lost, which meant 3,000 federal employees wouldn’t be getting their next paychecks.”
The IT team refused to accept defeat and continued troubleshooting. Their only recourse was to rebuild the 30 VMs from scratch. It took several hours, but it worked.
“We breathed a big sigh of relief, but we knew the system would crash again, and it did—four weeks later, just before Christmas. Again, some of us were home celebrating with our families, so we returned to the office and found that our backup software had failed again. I told my boss I wanted to replace it with Veeam, but he said we didn’t have the funds. That’s when I put my career on the line.”
The Project Manager worked with Veeam Software’s federal team to create a proof-of-concept test that overwhelmingly convinced his boss to replace legacy backup.
“We bet on Veeam, we won, and we’ve never looked back,” he said.
The government agency replaced legacy backup with Veeam® Backup & Replication™ to protect all IT systems. Veeam extends data protection to the cloud, verifies the recoverability of backups and recovers from them minutes, not hours like legacy backup. Veeam also supports regulatory compliance with the Privacy Act and the Federal Information Security Management Act (FISMA).
“Veeam and Azure are core components of our DR strategy,” said the Project Manager. “We know we can restore our systems fast in any emergency, giving us unparalleled peace of mind.”
Veeam backs up and replicates 32 TB across 400 physical servers and VMs from NetApp storage snapshots on-premises in a Cisco UCS FlexPod infrastructure. Veeam also sends backup copies to Azure. If there’s an emergency Veeam Direct Restore to Microsoft Azure lets the IT team restore on-prem physical and virtual workloads to Azure in minutes.
“Sending backup copies to the cloud wasn’t possible before we deployed Veeam,” the Project Manager said. “Being able to restore our entire onprem infrastructure to Azure from backups gives us a new level of disaster preparedness.”
Veeam enables additional “firsts” for the agency, and they’re all because of Veeam DataLabs™. This feature provides safe, isolated areas outside of the production environment to verify the recoverability of backups (SureBackup®), recover in minutes (Instant VM Recovery® and Instant File- Level Recovery) and test, develop and troubleshoot (On-Demand Sandbox™ for Storage Snapshots).
“Veeam provides many more features than we expected in a backup solution,” he said. “We’re really happy with its performance and trust it to protect our IT systems.”
More than 1,200 federal agencies trust Veeam to protect their critical systems. Veeam Software’s federal team understands and supports their needs, requirements and procurement processes. To simplify and streamline processes, Veeam partners with industry-leading hardware and software companies as well as system integrators. Veeam is fully certified and compliant to work with all agencies.
Compliance is a key requirement in government. Veeam supports agency compliance with regulations including the Privacy Act and FISMA. Agencies know their data is recoverable and available around the clock.
“We’re a success story because of Veeam,” he said. “I just wish we had deployed it sooner.”
To protect our customer’s privacy, Veeam Software has omitted its name from this story. The customer is a U.S. government agency that employs approximately 3,000 people worldwide.
Unless you’ve experienced a system crash firsthand, it can be difficult to understand the severity of the consequences. Now imagine facing that same crash twice—once over Thanksgiving and again over Christmas—when several members of the IT team have scheduled time off to be with family. That’s what happened to a federal agency, and the consequences hit close to home. The system supports payroll, so employees’ paychecks were in jeopardy.