This is a long-winded article, so bear with me here. I need to wake up any potential board of directors members that may be reading this. They need to be awake in order for them to approve air-gapped backups. But, that process takes time; more time than you usually get in front of a board, unless they have already fallen victim to ransomware or a wiper and survived it by luck. This means that these directors are now downright scared these data breaches will happen again. You see, ransomware or wipers, unlike lightning, is very likely to strike in the same spot twice.
My most current plan for air-gapped backups revolves around Veeam Backup & Replication v10. This update has a new feature that will help deliver air-gapped backups and immutability with object storage in the capacity tier of a Scale-out Backup Repository (SOBR). To do this, you do not need physical tapes or virtual tape libraries on specialized hard disk backup storage. All you need is AWS-compatible S3 storage that supports immutability. Technically, we are talking about Amazon S3 Object Lock in compliance mode.
This could be in a public cloud (i.e., AWS S3, Wasabi), at a service provider or an on-premises object storage solution that offers this. Now, how many of these will support Veeam Backup & Replication v10 Cloud Tier’s immutability feature? AWS is on board as well as Wasabi as far as I know. I am pretty sure many others will follow.
Microsoft Azure still needs to get its act together in this regard, but I am sure they will in the public cloud. In other locations, they have no real good offerings that are economically feasible in my opinion. Using Azure Stack Hub as a backup target with object storage doesn’t make financial sense unless Microsoft brings out specialized offerings.
We all have a role to play
If you haven’t heard of ransomware, you’ve probably been hiking around and living off the grid, but most of us are not doing that. Quite the opposite is true, we live and work in an ever more digital world; a digital world in which we have become as dependent on IT as we have on electricity. Yes, even hiking guides blog about their services and manage their bookings over the internet. This means that almost all of us are aware of the many cyber security risks out there. Too many of us have become victims of it.
As a result, you might make backups of your data. Maybe you even offload a copy of your data to “the cloud”. That is great, even if it is not always bullet proof.
In your business or place of work, you might just rely on your IT department or service provider and be happy with that. As an operational employee, this is fine, as long as you remember that you, personally, also have a role to play. You are a key element in protecting your business against malware, ransomware or other forms of cybercrime. Being aware of threats, recognizing them and not clicking on suspicious links is an example. Not falling for social engineering hacking is another.
However, when you’re in charge or otherwise responsible for the effective, efficient and secure operation of a business, you need to be a bit more concerned and involved when it comes to data protection. Merely having backups and hoping that someone will take care of things is not enough.
Have a plan
You not only need to have a plan A. You also need to have a plan B. By this, I don’t mean that you should be working on selling outdoor gear online if your business plan to be the best accounting firm for medium-sized businesses fails. Regardless of what you do, you require competent, reliable and ethical management practices. You most certainly do not want to end up like the woman who’s become the perfect meme for people discussing backups:
Don’t make bold
statements that you cannot back up. Be honest. Do right by your employees and
customers. I’m still baffled by the
For some of us, plan B is our job. Heck, I often have a plan C. What can I say? When the chips are down, I need options. Plural. One is none and two is one. That is the mantra.
Test your plans and assumptions
Also, try not to end up as the poster child for cyber security awareness. We all remember the frighting NotPetya attacks on Merck, Maersk, FedEx and others.
The stories after-the-fact make for fascinating reading. Especially on a winter evening after a nice dinner with a glass of wine in front of the fire place. What helps make these examples popular are the “happy endings”, the companies that pulled through. They did so by heroic effort, tenacity and good old luck. There are businesses that have been wiped out by similar or even less severe events. Those CEOs and CIOs are generally not interviewed about their visionary take on cyber security.
I am actually always a bit concerned when I see how important of a factor that luck is in most “success” stories. It is disconcerting to see that it is not just a windfall that helped things go faster or more smoothly, but luck is often a key element in being able to recover at all. Luck doesn’t last. Sooner or later we all run out of luck.
As digitalization becomes ever more embedded in every human endeavor, you can easily see how this will only get worse. When we have self-sailing ships, self-driving cars, surgical robots, smart cities, autonomous armed drones and AI-based justice systems, the threat of cybercrime becomes a lot scarier. In a near 100% global digital economy, driven by IoT, robotics and autonomous systems, the impact of data loss and unavailable services skyrockets to unseen levels. The damage that cybercrime and warfare can do will be exponentially higher than they are now. It will happen. If it is a systemic attack that’s going after infrastructure, transportation, economy, healthcare and emergency services simultaneously, things go downhill very fast. I’m pretty sure we will see some memorable disasters, probably some of biblical proportions.
Everyone has a plan until they get kicked in the face
Do you have a plan? Do you know from A to Z what to do? What about the things you don’t know about? The scenarios you could not even dream up, let alone mitigate? Can you really deal with a planned and targeted attack on your business?
The most sobering fact is that big names like Merck and Maersk, in the end, were just collateral damage. They were not the focus of a planned customized and targeted attack at all. All these companies thought they had at least some sort of a decent plan to deal with such an event. Their catastrophic encounter with NotPetya was just a spillover from a cyber-attack of one nation state (Russia) on another (Ukraine). Note that I am not calling this cyber warfare. It cannot, as my buddy at The International Criminal Court told me, cavalierly be classified as war merely because nation states were involved.
NotPetya has shown the world how bad being a road kill can be. We can now imagine what a targeted attack could look like. The blast radius and fallout of a grand scale, coordinated systemic attack on our societies are a magnitude uglier than the road kill of NotPetya.
A targeted and well-executed systemic attack on your business is a scenario that very few like to even consider. Some who do are paralysed by fear which leads to self-induced denial of service-level obstructions to their own business. Others die slow, boring deaths in endless work groups, taskforces, uninspired change management and life sucking ITIL. Some play this game to escape having to really deal with the threats at hand and maintain the illusion that they are handling the situation and therefore avoid blame. The above quite often leads to less action as the avoidance of failure is declared to be success and progress. All of this happens while their baseline of patched, secured, agile, modernized and monitored environments only fall further behind over time.
Often you can’t blame the employees, contractors or consultants for this behavior. They act that way because that is what is rewarded, or at least, not punished. They act in self-interest and for many, that salary or hourly wage depends on going with the flow. Such organizations are run by fear, authority, conformity and appeasement. Instead of creating a safe place of work where employees are not afraid to speak up and contribute to addressing real issues, they have created a dysfunctional family. Such a family will not be able to survive hardship in a manner in which they will have a bright and prosperous future. Paper is very willing, and will accept explanations, excuses and even downright lies without objection. Paper is a lot more willing than an engineer or technical architect that sees, often in real time, what’s right or what doesn’t work.
The above is bad enough in good times. It is deadly in hard times. Sugarcoating, spin-doctoring and blame-gaming doesn’t cut it when you look up and “behold a pale horse, and his name that sat on him was Death, and Hell followed with him.” Because that is what you face when the grim reaper of ransomware or wipers comes around. Okay, I got carried away with some biblical theatrics here. But do note that when a wiper destroys your business it will look pretty grim!
What will save you is preparation and realism. Having a solid, proven plan, along with the skills and the ability to execute that plan when necessary. ITIL, change management and risk management doesn’t mean anything unless it helps you act fast, think on your feet and not slow down actual effective and efficient actions in the field. Any time and money spent on methodologies and processes that don’t provide you with a force multiplier in your time of need are worthless. You might as well spend all that money and effort on a company outing to Hawaii.
You, the herd and immunity
Now that I have your attention and, hopefully, have woken up the board, we can almost get to air-gapped backups. Bear with me, please.
So how does one plan for a thing that’s so scary and daunting? Well, that’s simple actually. But simple doesn’t automatically translate into easy.
You deal with what you can control. Your own behavior, your team’s actions, your project’s responsibilities, the ones of your business unit or department, right up to your consortium is the level you operate at.
Everyone needs to act locally and think globally. That’s why you need a great organization to work in. If everyone is slinging mud, playing the blame game and looking out for number one whilst stepping on others, you don’t stand a chance at all.
As long as everyone does their best and acts in the local and global interest, we can achieve herd immunity. Sure, groups of individuals and entities will still fall victim to bad actors. However, combined protective measures will avoid systemic failure. This ensures the survival of our communities and economies. Critical infrastructure, transportation and emergency services will keep working. The economies will keep on running. There will be damage, both direct and collateral, but the system will cope, recover and be in a position to assist those who fell victim to attacks.
What you or your business gets out of doing your part is that you are well protected. This doesn’t guarantee survival but it sure does optimize your chances!
Backups, airgaps, restores, disaster recovery (DR) and business continuity
You probably want to know why I am talking about all this and how this is relevant to Veeam, you and me. Well, I am not the head of contingency planning at NATO or the Cyber Security Chair at Microsoft. I am not even acting at the CxO level. Maybe you are, but statistically speaking most of us are not. But I do have experience dealing with mediocre management, and l know that half of the time, we save ourselves without even realizing it.
Just like getting myself vaccinated helps with herd immunity while helping myself, protecting businesses helps protect the system on both a small and global scale. I am doing my part in this effort. This means I am always on the lookout for a better alternative way to achieve results. And that is where Veeam Backup & Replication v10 and immutability of your backup files in object storage comes into play.
I provide solutions at different levels. But whether my architecture is for a project, a business unit or the company as a whole, I do design for failure. Failure is not an option, it is guaranteed. Murphy’s law. Hence, I live by the mantra “one is none, two is one”. In the world of backups, this leads to the famous 3-2-1 rule. And, depending on the value of the data, I actually don’t even count the running copy in production.
Beyond the 3-2-1 rule
In these times of ransomware, going beyond the 3-2-1 rule means adding airgapped backup copies. But air-gapped backups most often mean tapes. Yes, they have a great cost/TB ratio, they have longevity, and they get the job done. Now, many of us have a love-hate relationship with tapes, if you have actually worked with them that is.
But in ransomware times, the off-site and off-line aspect of tapes have made it popular again. Many of us would love to get the air-gap capabilities of tape without having to use tape. Well, let me introduce you to Veeam Backup & Replication v10 that supports S3-compatible immutable storage at the file level in order to provide protection that formerly only air gapped tapes could deliver. This is a game changer.
The resurrection of tape backups is a reminder that we live in a time in which we are so frightened by ransomware that we literally look to the past for protection. We found this protection in tapes. It offers us a technology that fits the doctrine that guaranteed data recoverability is good, but we pay a price for this.
The issue with air-gapped tapes is that they come at a great operational cost. They reintroduce all the concerns for which we left tapes behind us. Hence, I have always been looking at better and more modern solutions to achieve the same results. Veeam Backup & Replication v10 support for immutable storage delivers exactly that. This is a game changer. The significance of this on our future designs for backup solutions is immense.
Air gapped or WORM (Write Once Read Many) tapes in a remote and secure location can provide the protection and recoverability we need. A copy of the data that ransomware or a hacker or cannot destroy (i.e., delete, overwrite, encrypt). What they cannot reach, access and touch, they cannot destroy or encrypt. Well, not unless the attack reaches the level at which they physically destroy those media in their respective location(s). That would mean a sophisticated, coordinated and distributed physical attack executed at-scale. There are following tape alternatives:
WORM is an option that is not only available via tape. There are other options based on disk storage, and there are disk-based solutions that offer WORM. One such example is
Immutable storage integrations with backups
What if your backup software can integrate with common, open standard object storage that has an immutable property on the file level? One that can be set but not unset until the chosen time period has expired, not even by the mother of all root accounts. That’s when you get the benefits or air-gapped copies without the need for tapes or specialized storage appliances.
Amazon S3 Object Lock in compliance mode offers exactly this for the duration of the retention period. Please note that S3 immutable storage does not mean you are tied to the public cloud of the hyper scalers. As long as you find object storage that can deliver S3 buckets with immutability implemented, you are good to go. This could be AWS, Wasabi or any other hosted, private/hybrid cloud or on-premises solution that supports it.
In the future I fully expect Microsoft Azure and others to implement this functionality as well in order to compete and offer the best possible options to their customers. The key here is that the immutability has to be on individual objects in the bucket or blob, not just at the bucket or blob level.
My aim here is two-fold. First of all, I want to wake people up and get them thinking about their data protection situation. Secondly, I want them to think about their role in the greater ecosystem when protecting their data. This is for both their own and their ecosystem’s benefit. I also wanted to introduce Veeam Backup & Replication v10 with support for immutability with object storage in the capacity tier of a SOBR. It is a subject matter I will spend more time on in 2020. I hope to do a technical blog about it later on, but for now, I hope I got both your attention and interest.