The Business Challenge
When BLR got its start in the mid-1970s, the company printed and mailed thousands of government compliance publications to customers. With the advent of online technology, BLR became a digital publishing company. Today BLR offers continually updated compliance information on 175 websites, which get 40,000 hits per second. Customers pay to access these websites in addition to online reports, events and training programs. Rightfully so, they expect 24x7 access. If they don’t have 24x7 access, customers may opt for a competing service.
“Technology has totally transformed our customers’ access to information, so we transformed how we deliver it—by putting almost everything online,” said John Burks, Director of Information Technology at BLR. “There was only one problem: Our legacy backup tool wasn’t able to adapt and grow to meet 24x7 availability requirements of our products and services.”
The legacy tool was designed to back up physical machines rather than virtual machines (VMs), which caused several issues. A full backup took 12 hours and could only be performed once a week because it put a strain on the production environment and slowed down multiprotocol label switching (MPLS) network traffic. Additionally, file-level recovery was slow, taking up to five hours because the correct tape had to be located and inventoried to find the right file. Disaster recovery (DR) was another issue because BLR wasn’t replicating VMs, and restoring a whole VM wasn’t possible with the legacy backup tool.
“The technology just wasn’t there and the backup vendor didn’t support Windows Server 2010 R2,” Burks explained. “The legacy tool didn’t have built-in deduplication and compression either, so backups required a lot of storage. This was particularly challenging because our data exploded by 400% from 2013 to 2015 due to organic business growth and company acquisitions.”
Burks said increased data and a substandard backup tool caused the IT team to lose sleep at night.
“I’d lie awake thinking how will we back up all this data in a centralized and timely manner? How will we store the backup data? How much more hardware should we purchase?”
To effectively handle the influx of data and modernize its data center, BLR moved its IT infrastructure to the Cisco UCS platform and rolled out Microsoft Hyper-V Server 2012 R2 on Windows Server 2012 R2.
“The backup vendor was only in beta for Hyper-V 2012 R2 support and therefore wasn’t supporting Windows Server 2012 R2 yet,” Burks said. “We were ahead of the backup vendor’s technology, which didn’t work—at all. We felt like we were living on the ‘bleeding edge.”
Burks went to Microsoft TechEd in 2014 to find a backup and recovery solution that would increase availability and enable BLR to meet service level agreements (SLAs). He watched a demonstration of Veeam® Backup & Replication™.
“I asked the Veeam rep if Veeam would really do what he said it would do. He said, ‘If it doesn’t, you come back to this booth and see me.’ I picked up my phone and arranged for a purchase order right away.”
Burks went back to the Veeam Software booth the following year and told the very same Veeam rep that Veeam Backup & Replication had saved the day for BLR several times. Burks also told the rep that BLR needed a better way to document and manage its Hyper-V backup infrastructure because its monitoring tool didn’t provide real-time reporting and interactive dashboards. BLR had no metrics in the virtual environment to document changes occurring in the data centers. The company faced another challenge with the monitoring tool. The tool delivered false-positive alerts about failed backups when they hadn’t failed. The Veeam rep demonstrated how Veeam ONE™ could help BLR take control of its Hyper-V backup infrastructure.
“Just like the year before, I picked up my phone and arranged for a purchase order right away,” Burks said. “I can tell you from experience that both Veeam solutions are outstanding. And, it all began with a Veeam rep looking me in the eye, establishing a relationship and making a promise that Veeam does what it says it will do.”
The Veeam Solution
Veeam Backup & Replication and Veeam ONE deliver 24x7 availability of BLR’s products and services, enabling the IT team to meets SLAs regardless of a 400% increase in data. Veeam solutions are designed specifically for virtualization, enabling backup, replication, recovery, monitoring, reporting and capacity planning for thousands of VMs.
In spite of 400% data growth, BLR can perform daily full backups because the backup window has been cut in half—from 12 hours to six hours with Veeam Backup & Replication. BLR is also able to replicate among its data centers, so if a VM fails or a data center burns down, BLR can restore data quickly.
“If you want worry-free availability, choose Veeam Backup & Replication,” Burks said. “Of all the solutions available, Veeam is the Ferrari sitting on the showroom floor. Coupled with Cisco UCS, our Hyper-V environment is rock solid.”
Daily full backups combined with Veeam’s fast recovery capabilities give BLR unmatched data availability. Burks said daily full backups give BLR additional recovery points, and more recovery points increase the availability of data, especially critical data on VMs running Microsoft SQL, Exchange and Windows servers. Whereas file-level recovery with the legacy backup tool took four to five hours, it only takes 20 to 30 seconds with Veeam. Full VM recovery, which wasn’t an option with the legacy tool, is just as fast with Veeam. Called Instant VM Recovery®, it lets BLR restart a failed VM from a regular backup quickly.
“One of our employees accidentally deleted a web server VM in production that runs our content management system because he thought it was a test VM,” Burks explained. “Using Instant VM Recovery, we recovered the web VM within 30 seconds without impacting 200 people—half our employee base—who access the content management system. That’s the value Veeam brings to us. We can recover a critical VM before our web alerting system pages our techs. It would have been impossible to recover that VM with the legacy backup tool, and that could have devastated our business.”
In addition to improving availability, Instant VM Recovery saved BLR $40,000 in hardware. Instead of setting up a separate test environment for web developers, BLR uses Instant VM Recovery to restore a VM from a backup if developers want to make a coding change to a web server during rollouts or testing.
Burks said BLR also saves money on backup storage because of Veeam’s built-in deduplication and compression. Most BLR backups have data compression ratios of 3:1, which reduced disk storage by 70%, saving the company $200,000 in hardware costs. Built-in deduplication and compression also decrease MPLS traffic during backup and replication.
“The other thing we love about Veeam is being able to mix storage because Veeam is storage-agnostic,” Burks said. “We made a strategic decision to buy cost-effective production storage and keep more backups on that storage before copying them to tape with Veeam, enabling us to save $30,000 in tape purchases each year.”
Based on its positive experience using Veeam Backup & Replication, BLR replaced its legacy monitoring tool with Veeam ONE. Burks said the primary reasons BLR installed
“We haven’t even begun to scratch the surface on what Veeam ONE will do for our business, but it already proved its value during the first two weeks of deployment,” Burks said.
“When you overhaul your IT infrastructure in four locations, your biggest task is not building up the systems; it’s documenting what you built,” Burks said. “Before Veeam ONE, our network administrators prepared Excel spreadsheets, and I prepared Visio drawings from spreadsheet data to share with the executive team. All together, we’d spend a week pulling together documentation. On the day we installed Veeam ONE, documenting our Hyper-V infrastructure took minutes. Veeam reports tell us what’s living where, how much CPU, memory and usage are available—saving us hundreds of prep hours each year. If I have to ask the CFO and CEO for more money before an acquisition, I show them system utilization, and it’s crystal clear how we are spending money. Veeam puts everything into common-sense form. I don’t have to spend time describing what they are looking at. It’s all in plain sight with Veeam.”
Veeam ONE gives BLR visibility into the Hyper-V environment that wasn’t available before, which helped the IT team identify an I/O issue.
“We didn’t realize we were having high IOPS on SQL backups,” Burks said. “We received alerts on performance thresholds because the backup window was growing large, so we looked deeper and decided to move SQL to flash drives, which cut the backup window in half.”
Veeam ONE lets BLR place metrics where there weren’t metrics before. For example, Veeam’s utilization metrics are extremely helpful when one of BLR’s business verticals rolls out a new product, especially in the Veeam report that tracks changes.
“If a new product is going to spike one of our websites, we need to schedule more resources for that VM,” Burks said. “Veeam ONE helps us prepare a game plan and review past game plans so we can do a better job next time. Veeam ONE also helps us meet SLAs. Before installing Veeam Backup & Replication and Veeam ONE, our website uptime was 80 to 85%. With Veeam ONE, we hit 99.9% for six consecutive months.”
Burks said Veeam’s technical support team complements its solutions.
“Each time one of our network administrators gets on the phone to call a vendor about a technical issue, we’re paying for his time because he could be working on other data center projects. We spend roughly four hours a week on the phone with a vendor, which translates into approximately $9,200 per year. We don’t waste that money with Veeam. Ninety percent of the time we use Veeam’s built-in knowledge base to answer our questions. When we do call Veeam, they pick up on the first ring and very quickly resolve our issue. We’ve called Veeam twice so far, and both situations involved a server issue, not a Veeam issue. Veeam is that good.”
Decreases recovery time by 99.8%
File-level recovery with the legacy backup tool took four hours; it takes 30 seconds with Veeam Backup & Replication. “We don’t worry about outages with Veeam because we can recover VMs running our websites without customers ever knowing,” Burks said. “There is no chance of data loss if you’re using Veeam. If you do lose data, it’s not because of Veeam. It’s because of human error.”
Reduces backup storage requirements by 70%
Most of BLR’s Veeam backups have data compression ratios of 3:1, which reduced disk storage by 70% even when data grew by 400%, saving $200,000 in hardware costs. Compressed and deduplicated backups enable BLR to keep more of them on disk storage before using Veeam to copy backups to tape, saving $30,000 per year in tape costs.
Optimizes performance and utilization of Hyper-V backup infrastructure
Veeam ONE’s interactive dashboards, metrics and real-time reporting help BLR document its virtual environment. “Never undervalue reporting in Veeam ONE, especially if your company is in hyper-growth mode like ours,” Burks said. “I can’t begin to imagine the amount of time I would spend documenting and forecasting growth without Veeam ONE. Preparing the capital budget will be significantly easier this year.”