As CIO for the City of New Orleans, I oversee IT for 5,000 employees who serve 400,000 residents and up to 20 million visitors each year. My remit includes police, fire, and emergency medical services — so ensuring that IT services are always available is a big responsibility on my shoulders.
But no organization is immune to disaster. The first significant incident that occurred during my tenure was a cyber-attack. It was unlucky but inevitable when a user with sysadmin rights clicked on a phishing link. The next thing we knew, our environment had been compromised and we watched in horror as our user workstations were accessed remotely and passwords reset. Our worst fears were confirmed when a ransom demand arrived.
We thought we were prepared, as we had seen other organizations in Louisiana fall victim, and we knew it was just a matter of time before it happened to us. The morning of the attack, we were on high alert, and as soon as we saw requests for elevation of credentials, we activated our ransomware protocol. Within two hours the state cyber defense team and the FBI were on the premises.
It was obvious to us that our best option for recovery was to rebuild our IT environment from the ground up, clean our data, and restore our systems; no negotiation with the criminals. I’m sorry to say that we weren’t a Veeam customer at the time. Unfortunately, the attack revealed just how much our legacy backup and restore solution was lacking. We were looking at months on months of work, and we weren’t confident that our existing tools could sanitize our data effectively enough to prevent reinfection. We knew that the PC environment of more than 2,500 desktops was compromised but did not know how many servers — two or 200 — were affected. Because of the existing backup solution’s clunky interfaces and poor performance, we were faced with a difficult and time-consuming recovery process.
With key city services at a halt, and with the City’s tax-collection season and Mardi Gras right round the corner, we had to get up and running ASAP. The City Mayor offered full support for our recovery plan, but the pressure was on, and a months-long rebuild simply wasn’t an option.
So, as we implemented replacement solutions and recovered our data, we searched for a new backup solution; we never wanted to be in this position again. After assessing the market, we knew it had to be Veeam. The simple user interface, the ability to back up data from disparate systems, the instant engagement from the Veeam rep… there was so much about Veeam that resonated with us immediately. We felt very comfortable that Veeam offered a highly effective solution that would be easy to use and powerful in offering us protection.
We got to work straight away, deploying new storage, cleansing our data, putting it back into production and implementing new backup policies, all supported by Veeam. We ended up getting all our services online again in less than 30 days, which is a huge improvement on the months we initially expected it to take.
Just 18 months later, we got to see how much difference using Veeam for data protection really makes. This time, it was Mother Nature that hit us hard. On August 29, 2021, Hurricane Ida knocked out the power to our data center. We shifted to diesel generators, which caused a fire to break out on the third floor of City Hall. All nola.gov sites went down, including ready.nola.gov — our emergency preparedness campaign. It was heartbreaking to see the damage the hurricane was wreaking on our beloved New Orleans, but we had to regroup and pull through as quickly as we could.
And this time we were able to recover much faster thanks to Veeam. We’d used it to automate key parts of the recovery process, allowing us to make better use of scarce resources and restore systems in just 48 hours. It was clear we’d learned a lot from the cyber-attack and Veeam empowered us to react faster and more efficiently. We rapidly stood up our secondary data center and got services back online. It was exciting to see Veeam work its magic and to be able to deliver for the city.
Our lesson learned is that it always pays to have a resilient, tried-and-tested DR strategy. Time spent prioritizing, categorizing, and tiering your data is never time wasted. Having a plan that you can trust is the best investment you can make, built on the most effective solutions in the market — in this case, Veeam.
As a government organization, negotiating with cyber-attackers isn’t an option for us. With Veeam, you don’t even need to consider that route. Our goal is to keep the city safe for every one of our 400,000 residents and the millions of people that visit us each year — and Veeam is helping us rise to that challenge.
After the trials of the cyber-attack, Hurricane Ida and the fire, I can say that the IT team got to have their champion moment. For example, after the fire, we had State Troopers moving equipment to our secondary data center, and we demonstrated to the city that it was worth dedicating those resources by returning to full operations just two days later. It was pretty exciting to prove that, given the right solutions and processes, a rockstar IT team delivers results.
Fingers crossed that’s the last disaster we’ll experience in a long while, but I can say hand on heart that we’re a leaner, more effective organization as a result of what we’ve been through. Veeam gives us a level of security and performance we didn’t have before, and you could say that with Veeam, I sleep better!
