Veeam Community Podcast

The Veeam Community Podcast is focused on giving the global virtualization community a new resource for connecting with industry experts, bloggers and peers, as well as for staying up to date on the latest industry news, developments and trends. Each 30-minute weekly episode of the Veeam Community Podcast will be available through RSS subscription and the Apple iTunes store. The podcast is hosted by Rick Vanover, a writer, blogger, VMware vExpert, VMware Certified Professional, Microsoft Certified Professional, Microsoft Certified IT Professional and a Microsoft Certified Systems Administrator. He currently works as Software Strategy Specialist at Veeam.

In this episode, host Rick Vanover (Twitter @RickVanover) welcomes both Edward Haletky (Twitter @Texiwill) of The Virtualization Practice and security instructor and trainer Tim Pierson to talk about penetration testing with virtual backups. This is a follow-up discussion from a recent webinar by Rick Vanover on using vPower in some creative ways.

Both Tim and Edward are on the hot seat for the “Three views from you” element of the podcast.

Rick: Hello and welcome the Veeam Community Podcast. I am your host, Rick Vanover. This is Episode 7, "Penetration Testing with Virtual Backups." Here we go. Today I'm really lucky. I've got two guests, Edward Haletky and Tim Pierson. How're you doing, guys?

Edward: Hey, there.

Tim: Just great.

Edward: How are you doing, Rick?

Rick: I'm good. I'm good. So, everybody knows Edward, he's Texiwill on Twitter and VMware Communities, and author of a number of security books. Edward, why don't you tell everybody about yourself?

Edward: Well, actually only one security book, it's "VMware, vSphere, and Virtual Infrastructure Security," and Tim actually was a contributing author on that book. He wrote our penetration testing chapter, which is very appropriate to this conversation.

I do have a new book coming out very soon. It will be "VMware ESX and ESXi in the Enterprise: Planning Your Virtual Environment," and that should be out any time now.

Rick: Original version, this "VMware ESX and ESXi in the Enterprise" is actually the second edition to "VMware ESX Server in the Enterprise."

Edward: "VMware ESX Server in the Enterprise," its subtitle was "Planning and Securing Your Virtual Environment."

Rick: That's it. That's it, okay.

Edward: So yeah, there are two, but I'm concentrating only on the vSphere one right now because I took all the information from that one and put it into the bigger book.

Rick: Cool. Tim, so tell us a little bit about yourself? You're like a Twitter stalker you said. Other than that, tell us a little bit about yourself?

Tim: Well, basically, I've been an instructor for about 25 years. I love to teach all of the different technologies. I'm a technical trainer. I won the Instructor of the Year from EC Council last year, that was delivered at their Hacker Halted Conference. I deliver at all of the conferences all over the world.

Rick: Oh, cool.

Tim: Gosh, I was just in Singapore. I was in Kuala Lumpur. I was in Miami. I was in Sri Lanka, just different places and this is just kind of my passion, to see if I can break into something that somebody thinks is safe.

Rick: Oh, yeah, cool.

Edward: Actually, Tim and I worked together on quite a few things, and we taught a secure coding class together a few times in Europe, which was really interesting. We surprised a bank with some ideas.

Rick: Well and that's really relevant to what we're going to talk about, because we had a financial services organization give us an idea from one of our systems engineers of a way to use vPower's Sure Backup capability. Let me take just a minute and explain what that is.

With vPower, one of the categories of functionality is something called Sure Backup. Sure Backup will take your virtual machine backup - in the case of Veeam it's effectively a snapshot based backup of the virtual machine in a running crash consistent state - and then that backup, we're capable of powering that on, on a special NFS data store that's a provision of the host to an ESXi host, a single host, for either automated testing of your backup file or an on-demand sandbox capacity. Those virtual machines, when they're powered up, they're configured to be behind a network proxy appliance, effectively a Linux firewall that just boots up with a running config of traffic in only. There's no traffic out.

So that capability, we had a customer say, "That's a great environment for me to do my penetration testing on the backup of XYZ multi-tier application." So we were talking a little bit about that, and I couldn't think of two better people to come on and talk about that. So, guys, just generally, what do you think of that?

Tim: Well, we basically had this idea, and as a matter of fact I had talked to some other vendors about getting some of their technology to demonstrate it and one thing led to another and I just never did do it. But basically, what the concept is, is if you go up to a systems administrator and ask the systems administrator, "I'm going to throw everything but the kitchen sink at your DMZ." Well, most systems administrators, their machines are kind of like their kids. They want to try and protect them. They want to, "Okay. What are you going to do?" "Well, when was the last time the hacker told you what he was going to do?" "Well, I don't know." "I'm just going to throw everything I can." "I don't think so."

Edward: A lot of sys admins and security people won't let you, even if you're doing white hat penetration testing, or red team or blue team, or whatever it is, they will not allow you to run the destructive tests at all.

Tim: Exactly.

Edward: There are so many of them. There are so many destructive tests because you're running in a live environment. So the concept of running in a sure backup off your NFS server is a great idea. Tim and I actually put this together . . . we started thinking about this 2005, 2006 time frame when ESX 2.5 was around, and were thinking about using P2V tools to do the exact same thing.

But you have two problems with the approach that Sure Backup takes, that I can see right off, just thinking about Sure Backup and how it works and the fact that it's using an NFS server. First is, is you're using a firewall. You don't want to have it firewalled. You want it actually on a net mix segment that doesn't go anywhere, no firewalls, no switches, just the host and nothing else. So if there's a firewall, you're going to end up penetration testing through it.

Rick: So in VMware talk, let's kind of tweak the practice a little bit. So let's take that Sure Backup VM, let's disconnect it from that port group that is behind the firewall, let's put our penetration mechanism somehow on there. If it's virtual, great. If it needs to be hardwired on a physical port segment, then we get the host on that. So we take that as kind of avoiding that firewall, either virtual to virtual or physical to virtual through the physical infrastructure.

Edward: Well, there are a couple of different approaches here and that is, is that, "What are you wanting to penetration test?"

Rick: Well, that's the proverbial, "It depends."

Edward: Well, it actually doesn't.

Tim: Let me get back to what I was saying earlier to the systems administrator. When you're asking him these questions wanting him to know what you're going to do, he's not going to basically give you permission to do anything that you want. "Well, what if I took a copy of your entire DMZ and replicated it exactly as it was." "Well, knock yourself out. Let me know what happens." Then when you find the thing that it did hiccup on, you can, with permission by the systems administrator, try just that thing. Is it perfect? No, it's not perfect. Is it close to perfect? It's about as close as you're going to get.

Edward: So remember, the attackers, when you do penetration testing, you're not actually penetration testing from within your own network. You're actually penetration testing the entire bastion level of security. In what we're talking about, you're talking about the firewall to the DMZ, the VMs or the physical host within the DMZ, the firewall to the backend network if there is one. You're testing all the way through those layers, so just restoring the one application doesn't help us. We need all those other pieces together.

So putting it in VMware speak, if I have vShield Edge, acting as the firewall to my DMZ, I then have DMZ VMs and I'm running vShield app within that and vShield endpoint on top of all my VMs, and then another vShield Edge to give me my internal network. Right? If I'm layering things like that, doing what you're talking about with Sure Backup, perfect, because I just restore it all, put it on a totally separate set of what I call, and some of the stuff I've been talking about lately, is a secure cluster of ESX hosts. That once you put the stuff in there, I mean, I'm not talking about using NFS, because NFS, there are a bunch of attacks that are timing attacks, and if you're running 100 VMs, your NFS server may crater.

Rick: Yeah, true. Sure.

Edward: So you restore it to a security location. Then you basically cut the wires. Then once you cut the wires to the outside world, you bring in your attack tools like you would normally do it, which is you would go off and do . . . I mean, what are the steps, Tim, to penetration tests?

Tim: You basically do all the things you do, your port scan, your banner grab, your penetration, all of the things you would normally do to try and get in, fuzzing tools, which is usually the most destructive, which will usually cause the thing hiccup. You have permission now. You have carte blanche access to throw everything you want to at it because you want it to fail, because when it fails, you can try and replicate that back on the physical environment, with permission. That's the beautiful takeaway on this.

Edward: Yeah, not only that, the key thing here is that a lot of these penetration tools don't just attack a single app. They'll do side channel attacks. They may attack DHCP, they may attack DNS. They attack AD and eventually get to the app. So there are a lot of attacks that aren't just, "Let's pound on the app." No. "Let's pound on a part of the app or something . . ."

Tim: For what we're talking about, yes, that's true.

Edward: So just restoring the app, its middle tier, its middleware, and its database and attacking them, well, you're attacking . . . the type of security you're testing is really what a developer wants to test. You're talking about did the code get developed properly? Was the security stance of that set of VMs set up properly? But you're not really doing a full penetration test. You're only doing a part of it because everything is interrelated.

Tim: So why do you say you're not doing a full penetration test? Because it's not in a production environment?

Edward: No. No, it's because, for example, Tim and I talk about attacking ESX all the time, which becomes a part of your penetration test. So if I'm restoring the VMs and just attacking the VMs and ignoring ESX, I'll guarantee you no hacker is going to do that.

Rick: In vCenter, for that matter.

Edward: In vCenter for that matter.

Tim: Because the prize is much bigger.

Edward: And not only that, you need to worry about, "Hey, side channel attacks." You've got to worry about, if I attack DNS, did that just suddenly let me into the app because the app was responding to it incorrectly? What about your certificate server? Are you doing revocation look up? What happens if you can't reach the CRL? All these things become part of the penetration test.

Rick: So it's not just one VM. It's the entire infrastructure, EXS, vCenter, on that [inaudible 11:42], and then say your active directory and your Windows apps, web database, app server, all that stuff, the whole thing.

Edward: The whole thing and what you're talking about with single apps, yes, it's vital to do that, absolutely. But I look at that as more of a developer view of the world. It's a very narrow scope, and that's actually not what the hackers are going to look at.

Rick: True, true. So in the example of doing penetration testing on Sure Backup, you'd really need that full slice, and if you have the capacity for a secure cluster so that you could attack ESX, assuming that it's configured exactly the same as the production environment, then again, that's just one step closer to be as fully representative as possible for that function. Would you guys at least agree on that?

Edward: Absolutely.

Tim: Exactly.

Rick: Cool, cool.

Edward: Any security lab that does this is going to want to duplicate your networking. I mean, all of the multiple physical layers of physical switching. If you're using Blade, you've got the physical virtual network, they're going to want to duplicate that, as it is. Then they're going to want to duplicate the ESX host or the ESXi host. Then they're going to want to put the VMs in the right place inside there and the distributor, virtual switching and everything. The vast majority of penetration tests today are over the network, aren't they?

Tim: Yes, they are. Basically, even if you're wanting to penetrate Cisco devices, there's a technology called GNS3 where you can actually virtualize it and put it onto a machine, and run the actual IOS that Cisco is running. I mean you can replicate it right down to exactly what you have.

Edward: In the multiple layers and their appropriate uplinks.

Rick: Edward, what was the second thing you saw as a problem with the idea?

Edward: Well, the other thing I saw with NFS is, if you're going to restore a full environment, one or two or three VMs, I don't see a problem. But if you're restoring 100, a 1,000 using your NFS server, I think that's just going to crush it.

Rick: Well, I don't think 100 or 1,000 is kind of within the use case of Sure Backup or vPower. I really don't think it is.

Tim: I was thinking more on the lines of a DMZ, is how I had envisioned it.

Rick: Or some logical unit of work, right.

Tim: Exactly.

Rick: Maybe pick a number, 20 VMs.

Tim: . . . [SS] . . . department or whatever. And replicate it as close as you can without them having to attack the real physical environment. I mean, you're home free and your systems administrators are going to love you.

Edward: Mm-hmm.

Rick: When you're doing these types of, not so much your day-to-day scan, but anything that's more involved, like the white hat test, that you mentioned, you don't actually know if there's a problem until you actually try to inject some bad data. Is that right?

Tim: Exactly.

Edward: And the other thing is, is that you don't want it tied to your backup server in any way. As I said, you want to restore it to a secure lab and then cut the wires.

Rick: In Veeam, talk you probably want multiple, to really take it to the ultimate level, we'd have to do additional separation at the backup server level.

Edward: You'd have to replicate.

Rick: A separate backup server using the backup file from the first server.

Tim: How about if you just unplugged the wire?

Edward: Well, if you're using your backup server, the thing running NFS, Tim, has to be available to the ESX host, which means it's physically on the network.

Tim: Oh, okay. So you're talking about still being shared by the same one, okay. I was actually talking about a complete replicated environment that . . .

Rick: Well, yeah, that would do it too as well, Tim, but the use case I had in my head was that people don't have to provision additional storage by using the Sure Backup because we're running from the backup file that's already on disk. The only additional storage is effectively the running config, the page file that gets created on the VMs, and whatever gets added additionally. We don't have to do it like a bulk clone of all these VMs, which is a huge amount of storage.

Tim: It's almost like a [inaudible 15:57].

Rick: From the backup, yeah, you could call it that. Yeah.

Edward: That means that Veeam Sure Backup vPower becomes in scope for such a test.

Rick: Well, you know what? If it's protecting stuff already, in most situations it would already be in scope. If it stores, transmits, or processes the stuff, it's already there, in a sense.

Edward: Mm-hmm.

Rick: Well, hey, guys. So we got a lot of details here for people to think about, and there's no blanket recommendation, but Tim and Ed have done a great job of giving us things to think about pretty much to carve up an environment that will provide you an incrementally more representative platform or secure cluster, secure environment using a couple of different technologies for things like penetration testing. I wouldn't say this is all necessary for everyday scanning because just general purpose scanning I would just say you go on the live environment as is, because that's not really that obtrusive, unless you think differently?

Edward: Tim, you know some scanners that are pretty nasty.

Tim: I know some scanners that'll be very detrimental. I would do everything on the replicated environment. It just depends on how nervous, I guess, your systems administrator is.

Rick: Well, true or how sensitive the environments are.

Edward: Yeah. There are scanners that will take down whole networks. So you want to be careful what you're running anywhere.

Rick: Right.

Edward: As Tim said, it really depends on how risk averse your admins are.

Tim: When you do a port scan to do a sync, sync ack, and don't respond with the ack, and you do it on all 65,000 ports, you've basically brought down that machine. So it's possible to do a port scan and cause damage. See what I mean?

Rick: Oh, yeah.

Tim: You have no ports to give up.

Edward: Well, not only that, what about if server doesn't respond to a scan well and suddenly shuts off, like vMotion used to? If you scanned a vMotion port on an ESX server at one time – this was fixed recently – vMotion would no longer work. VMotion has become a critical component of just about every infrastructure I know or every environment for saving after hour work, or just for contention balancing, on DRS, in dynamic resource load balancing of that type. If you turn off a critical component because of a scan, well, you're definitely not going to want to do that inside your real environment. In a test environment, absolutely.

Rick: I agree. My objective with this podcast is you just give people additional tools, and in the case of the Veeam world, they may already have this capability with vPower that they can maybe provision these other environments, take these additional extra steps to incrementally make the environment more capable and better for things like this. So that's really good stuff.

I have a question though. Or I should say, not really a question, but just kind of a lead for everyone else. Edward and Tim, both, work for the Virtualization Security Roundtable podcast hosted on TalkShoe. Ed, Tim, why don't you tell everybody a little about your podcast?

Tim: Okay, Ed, it's your podcast.

Edward: It's actually my podcast. It's actually interesting. Tim's involvement's kind of unique in that he comes on whenever he can by himself, but actually quite a few times when he's off teaching a security class, specifically on virtualization, the whole class will join in on the podcast when they're timed up right, and it's actually pretty cool to have the whole class there.

But the podcast is generally looking at virtualization and cloud security issues across the board. It's hypervisor agnostic. Yes, we do talk a fair amount about VMware because they happen to be the largest selling hypervisor out there. But we're across everything that's related to virtualization and cloud.

Rick: I like your podcast. I do listen to it. I subscribe to it on my phone, because, especially when I was just the administrator, the end user, every now and then you would find a nugget of information that's an easy fix. Just imagine that Ed is standing behind you as you're configuring a server, and then if you have that fear behind you, you can do little things to make it better. So that's the way I like to look at it.

Hey, so Ed or Tim, I have three questions, kind of the gimmick of the podcast. I'm curious, who wants to take them? Ed might already know what they are.

Edward: Tim

Tim: He's the savant, so . . .

Rick: Oh, okay. All right.

Ed: Which type of savant is that?

Tim: That's the good one.

Rick: The good one, nice.

Ed: Well, I know what, we'll alternate. Tim first.

Rick: No, that means you both get them. All right. So I'll start with Tim. Hey, Tim, in your professional work experience, looking into the past, you know not right now, but anytime before, what was the most interesting area of technology or story that you can share with us about something you've done technology wise in your past?

Tim: Virtualization or not virtualization?

Rick: Technology. I don't know, working on cars, phones, whatever.

Tim: When I was little I was always taken aback by my dad. He would always pull me by the back of the shirt and say, "Don't you take that stuff apart." I would go over to people's house and I'd have the back of their TV off and their VCR taken apart. That didn't mean I didn't necessarily put it back together, but I'd have it all pulled all apart. He be there cussing and trying to put it all back. "I told you not to take this apart."

I was the type of person who was just curious. I wanted to see how things work. I used to tell Ed all the time . . . he would tell me, "Don't you know that?" I said, "Ed, I know how to break things. I don't know how to fix things necessarily."

I think my biggest piece of the technology that I find most fascinating is when things hiccup, as I say, when things break or where their breaking point is.

Rick: No, that's cool. That's a pretty cool story. I can relate a little bit to that, maybe not as dramatic, but a good way of telling it.

Hey, Ed, now for you, what do you have going on in your past or what have you had going on in your past that was the most interesting or, I don't want to say a fish tale, or a good story about technology in general in your past that you can share?

Edward: Several things, my love of technology actually all goes back to when I was a little kid, I mean really little. I have a vivid imagination. That probably helps in this particular and I actually keep people up at night, I think.

Tim: And you still do.

Edward: Specifically, Tim and a few others, but actually it all goes back to it wasn't my first programming thing, which was 1,000 lines in a day, but lost a friend over that one. No it was actually before that. I was even younger. I would play around with whatever I found in the yard as if it was this big engine. I always tried to make it better, gears, and levers, and rocks, they all become part of the this.

Tim: He would soup it up.

Edward: Yeah, I'd soup up a tree stump to make it this huge engine. I've always been intrigued by technology from that aspect, that's why I do what I do.

Rick: Cool, cool.

Edward: From a technological perspective, though, I mean, that's way back then. Why I'm on doing what I do now is that I did a bunch of programming a long time ago, and I had to write that program across 30 different systems. I always wanted one system that just did everything.

Rick: Now you probably don't want that though, do you?

Edward: Oh, actually now, I'm approaching what I have. I have just one system that runs multiple operating systems, and I'm happy with that.

Rick: Okay.

Edward: Now this was many, many, many moons ago before virtualization was a gleam in anybody's eye, except IBM, but we don't talk about that.

Rick: So, Tim, what's going on in your current technology practice right now that you're most interested in?

Tim: Well, I have a new technology that I'm developing with another good friend of mine that is pretty much going to revolutionize the way that we deliver classes. It's called Flexible Presenter. It's basically going to give you the capability to take a class anywhere, anytime, and see the person that's sitting right next to you, and that person could be right there on the other side of the world. I've shown it to Ed. All of this stuff is going to run in the cloud.

So the part about security, where most of the time the U.S. government has this directive called the 8570 Directive, and they're not getting a lot of traction on it because their people don't have a place to practice. They don't have a playground. So consequently, they would go to Uncle Sam and say, "We're going to put this malware in your environment." He says, "I don't think so." Go back home to you wife and say, "We're going t to build this thing." "Oh, we've got diapers to buy, we're not building this thing." Well, then it basically remains, I guess for the lack of a better word, unplayed with. They don't have a chance to play with the tools. They don't have a chance to really get good with the tools. The only way that you're going to catch a hacker is to be as good as a hacker and know the tools that they are using. So I've developed an environment so that you can basically RVP into, and you could use all of those tools without ever having a fear of touching your physical environment. It never sneaks back to you. It basically has step through labs. It has video based instruction. It has testing. It's really pretty cool.

Rick: That's cool, because, yeah, that's the hardest thing. Not just security, but even just learning how to provision infrastructure, you don't know what mass changes on your storage here are going to do unless you do it. Right?

Tim: Exactly.

Rick: Cool. So, Ed, what's going on right now that you're most interested in, in your technology practice, virtualization or not?

Edward: That's a tough one. I actually have my fingers in so many different pies. The most interesting thing that I'm working on is actually trying to translate the current thinking about virtualization security and cloud security into an organizable, recognizable stack with a list of all the threats and protections you can put into place. So taking what's in my book, adding all the new stuff since that book was published, and putting it together as something people can use to go further and understand what's going on, more education.

Rick: Sure.

Edward: How to fix the problems, not just, "Hey, here they are."

Rick: Right.

Tim: I break it. He fixes it.

Edward: Tim goes, "Here they are." And I go, "Here's how you fix them." No, actually Tim's class is all about fixing them too. But this one's, the approach is a more architectural level.

Rick: What are you doing there, Tim? Was that your dog?

Tim: That was my dog.

Rick: Okay. Sorry. We've got video on this call, guys. I didn't know what was going on, but it is his dog. Okay.

Edward: Now, Tim, I don't have video from you. You're frozen.

Rick: Looking into the future, what are you most excited for technology wise? What do you see coming that you really think you're going to latch on to in the future?

Tim: I'd say the cloud. I mean, being able to use the cloud with my Flexible Presenter product. I think that's going to be the big thing where I don't have to worry about security. If somebody breaks in, "Okay. Well, big deal. Just reboot the thing and start again." It's not going to be that big a deal to me, whereas businesses if you break in to the cloud, that could be critical data. I'm just basically using tests and scenario data to use the cloud. So I think that's a perfect ways of introducing the cloud to people.

Rick: Okay. So, Edward, what do you see looking into the future, technology and general, most exciting, most interesting in your view?

Edward: Cloud forensics.

Rick: Hmm.

Tim: That's true too.

Edward: In dealing with the virtual environment as part of the cloud, multiple layers of the cloud. What do you do to do forensics? The techniques that are out there right now are actually overlooking some serious capability within the virtual environment. By serious, it's more, "Hey, that would make my life as a digital forensics engineer or scientist a whole lot easier."

Tim: Because you can now save a memory footprint of the server when you shut it down. We can wrap the switch to shut it down and save a snapshot of the memory footprint. Because basically what happens when you come in and your servers are acting all squirrelly, what are you going to do? "Well, let's go take a look at our log." No. You're going to reboot the stupid thing. You just lost all of your evidence.

Rick: Right, right, right. Well then, the next thing is a compute log, right? We've got disks, we've got memory, what's next? And we have network.

Edward: You can get compute log now.

Rick: Really? Cool.

Edward: You can actually get the compute log, you can get the memory, you can get the disk, and you can get I/O without anybody inside that VM knowing you're doing it.

Rick: Underneath the platform. Then, how do you protect that? Because if you're running a cloud that's somebody else's data, does the cloud provider have rights to it? There are a lot of questions around that.

Edward: Actually, what's really interesting is whatever you can do in forensics, a hacker or an attacker can do as well, but from a destructive or data gathering approach as well. So, from a forensics perspective, doing forensics analysis and research into a virtual environment within the cloud, that's really a very interesting to me because I think its such a new space. You are talking about volumes of data that most law enforcement, there is a few that do, will not be touching any time soon. They want to reduce that amount down so it's a manageable bundle. But if you are dealing with terabytes of data, I just heard of an Exabyte tape drive, tape library, so there could be Exabyte's of data out there soon. Forensics on that becomes extremely painful so GRC is going to be a very important approach to doing forensics. How much {inaudible 31:50}? How much logging? Whats missing? How do we change it? How do we do secure deletes? How do we do secure rights? These are the issues that to me, are very important. They are the exciting thing because no one has really done it yet. They are getting there.

Rick: Wow. That's awesome. I am not really the person to go and be a thought leader and that, but I definitely have interest in in because there is a huge opportunity for all that stuff. Those are great responses guys. We are going to wrap up the podcast here. If anybody, any of the listeners have any questions about the podcast, comments, observations, you can send an email to podcast@veeam.com. And Tim, Ed, thanks for being on the show.

Tim: Thanks for having us.

Edward: Thanks for having us.

Rick: You're welcome guys. Cheers.