Veeam Community Podcast

The Veeam Community Podcast is focused on giving the global virtualization community a new resource for connecting with industry experts, bloggers and peers, as well as for staying up to date on the latest industry news, developments and trends. Each 30-minute weekly episode of the Veeam Community Podcast will be available through RSS subscription and the Apple iTunes store. The podcast is hosted by Rick Vanover, a writer, blogger, VMware vExpert, VMware Certified Professional, Microsoft Certified Professional, Microsoft Certified IT Professional and a Microsoft Certified Systems Administrator. He currently works as Software Strategy Specialist at Veeam.

In this episode, host Rick Vanover (Twitter @RickVanover) welcomes Chris Evans (Twitter @ChrisMEvans) and Doug Carson (Twitter @Douglas_Carson) to discuss VAAI storage for virtualization. Chris writes The Storage Architect blog and consults on various IT-related topics, including virtualization and is based in the UK. Doug is a systems integrator based in the UK as well.

Chris is on the hot seat for “Three views from you.”

Rick: Hello and welcome to the Veeam Community Podcast. I am your host, Rick Vanover. This is Episode 9, "Virtualization Storage with VAAI and other Smarts on Disk". Here we go. Today our guests are Chris Evans and Doug Carson. How you doing, Chris?

Chris: I'm good. Thanks very much for inviting me onto your podcast.

Rick: Hey, not a problem. Doug is with me in the co-pilot seat. How are you doing Doug?

Doug: I'm doing fine. Thanks for having me again.

Rick: Great. I appreciate you guys taking some time to talk with me today. One of the things we want to talk about is virtualization storage. I can't think of anybody better than Chris Evans. He's in that small circle of people that if I want some good, honest opinions on storage, that's who I'm going to go to. Chris writes the blog at the Storage Architect, and he covers a lot of virtualization stuff as well. He has a broad consulting practice. Chris, why don't you tell everybody a little bit about you, what you do and where you're from and all that kind of stuff.

Chris: Okay, thanks, Rick. I'm based out of the UK, and, as you said, I write a blog at the StorageArchitect.com. It originally covered storage, but it does cover virtualization now and other things occasionally if I go off base and start talking about other stuff. In my day job, because obviously I can't afford to blog full time, I run a consultancy with a number of colleagues. It's a company called Langton Blue. We look at storage, but also at the wider IT issues, solving problems for people, that's both technology and process.

Rick: Cool. If you haven't seen the Storage Architect blog, I really encourage everybody to check it out. It's good stuff. Storage is so critical to virtualization. It might be understated. Sometimes you can probably get away with it in smaller organizations, but as you scale up and out, it's definitely a critical part of it. Chris and I have had a number of discussions over the years about VAAI. Now there are two other series of features that are coming for storage that are going to make it a lot easier. Chris and I one time were talking once about, "Well, it would be kind of cool if VAAI did this and VAAI did that." Chris had the best idea ever with secure defrag.

Chris: Yeah, thanks for that Rick. Here's the thinking. When you do defrag, you move blocks of data around, and a lot of the times those blocks are moved logically. You copy the data, but you don't delete the old blocks. Whilst that's fine when you can keep track of them in say a defrag situation, it isn't good if somebody took that disk and wanted to scan it. They could still find potentially old copies of data. My idea was to allow you to wipe those old blocks clean by using a command that would just tell the array to write zeroes over it. It would be much more secure than today's defrag system but wouldn't take a lot of extra CPU and I/O to actually achieve. That was the basis of it.

Rick: That I think would be great, and that's the type of thing that we'd have to hook into VMware tools and all of that kind of stuff to know where to look. That's just one example of our pub talk, if you will, of great ideas for this and that, and he had the best one. We're going to talk a little bit about storage today specifically for virtualization. There are a lot of new things coming out. VAAI, as we know, for certain storage platforms is going to be good stuff. We've got sub-LUN tiering available in a number of storage products now. Looking into the future, we're getting some glimpses of what Storage DRS is going to be like. Doug and I were on Twitter talking yesterday, and we're like, "That might be a problem, something like Storage DRS and sub-LUN tiering." I don't want to say tripping over each other, but guys, what do you think of that potentially?

Doug: My thought was I'd just throw a couple of things out about Storage DRS, and then I started thinking about multiple tiering and whether or not, as you said, the two can step on each other. That was the thought. Do you really need Storage DRS in terms of moving VMs to other data stores to reduce I/O and stuff like that to help out? My view would be that multiple tiering would deal with that. So I'm just curious as to what Chris' thoughts about that would be.

Chris: Let's go into two thoughts then. First of all, the idea of what the sub-LUN tiering would be, and clearly the idea of that is to place individual components of a LUN onto different tiers of storage, i.e., different performance tiers of storage. If you have a hot block within a LUN, that block will be better I/O quality without having to dedicate that whole LUN to that tier of storage. You can be much more granular and much more efficient with your I/O responses. That's great in an environment where it auto balances out and the host doesn't need to know what's going on. Clearly, DRS would come in and start saying, "I need to set the I/O policy based on what I think this disk can now do." That's a continually changing scenario. I haven't seen a great deal of what DRS will do yet. I don't know whether you guys can answer to what you think DRS will do exactly, but if it's that performance issue, then potentially you could find the two fighting against each other.

Rick: The fundamental thing about Storage DRS, and I'll admit that I need to review some of the materials. They had some previews at VMworld and stuff. A unit is a virtual machine. The virtual machine has individual virtual disks, which would be on one or more data storage which are on a LUN. Those units of measure include the virtual machine and then the individual VMBKs and their individual associated latencies. So if something like sub LUN tiering on a storage processor can pick that up and move around those hot spots to keep that unit of measure on the virtual machine good, as far as overall latency . . . by no means is sub-LUN tuning mainstream and by no means is VAAI mainstream and by no means is Storage DRS available yet. We're kind of going theoretical, but these are the types of things we want to think about before we load up on these technologies and just run with them. I think that individual unit, the virtual machine disk latency measure per data store, in terms of that virtual machine, that's going to be the differentiator, the hand-off point between the two technologies. Would you agree with that, guys?

Chris: That makes sense. The other thing that I would be interested to find out is whether DRS will look at things like the paths to the device. If you've got multi-path I/O to the device, will it load balance across those paths as well as moving data around based on latency?

Rick: Well, that depends on the storage protocol and storage product in use. That's the correct answer. Other than that, there are some yes's and some no's in the mix. I think these are good features coming out, and I'll admit I've not used a VAAI supported storage array yet as far as the right version of vSphere, the right firmware on the product and all that. Chris, have you?

Chris: No, I haven't and I'd really like to get into a position where I could. I think it's such a new technology I don't think it's widely deployed out there, and we know that a lot of the vendors have only recently started supporting it. Stephen Foskett did a great summary of that by rate type, and even though the vendors have only just supported it, it's obviously on the latest versions of code and a lot of people may not have put that anywhere other than into their lab environments.

Rick: Absolutely, because it's not a normal update in most situations in terms of vSphere because vSphere updates are very regular. We just had one recently. We have Update Manager, or you can do it by hand with the host update command. But storage firmware, generally speaking, if you have a product that's already in place, those don't get upgraded as frequently as operating systems and things like VMware. I think that the cycles are moving at different rates. If you have a new product, that's different. If everything's new, like a new segment of the data center or a whole new data center, it might be easier off. How about you, Doug? Have you used any end-to-end VAAI solutions yet?

Doug: I'm fortunate I have, actually. We've done a couple of lifetime solutions, which we've used which are [inaudible 09:20] some VAAI integration into it. So we've had a wee play with that over the last few weeks. Some quite nice things. It's good.

Rick: The one thing I would see as clearly a big help would be deploy template. How did that go? Or formatted data store?

Doug: Definitely. I mean I think maybe it's the first stage in terms of the storage vMotion story, if you can call it that. I think the ability to offload that kind of load instead of going through the ESX hosts to the [inaudible 09:54]. It certainly helps for things like Storage DRS, but also in things when you're deploying out VMs quite regularly or you're maybe doing virtual desktops and stuff like that.

Rick: Did you do any snapshots? Anything with snapshots in any of those environments?

Doug: No, they were very, very basic environments. Experience is still quiet short right now.

Rick: No, that's not a problem. That's telling it like it is. I like it.

Chris: It sounds like you've done a lot of functionality testing. So you've had a product. You've played with it. You've tested some of the features to see whether they work. My concern with some of these new features is how they'll scale and how they'll work in a large environment. So if you think of the fact that you've got the ability to push down a huge amount of I/O workload to the underlying hardware, it could be very easy for somebody to overload it. If that's a shared environment, that could be more of an issue than it would be not having those features.

Rick: Oh, wow. That's another metric beyond your normal IOPs is your CPU cycle on your storage processor. I think you could get away with it in a lot of modular storage and below, but I think if you, on the high end modular or even probably of course the storage vendors would want to tell us otherwise, but yeah, it's uncharted water for sure, Chris.

Doug: Especially when you think about multi-tenant and stuff like that. There are lots of guys who are doing different things, how that would affect different environments, stuff like that as well.

Chris: I think that touches on another point. That's a great point about the multi-tenant stuff. But now touching on that point about the ability for us to integrate that hardware and do an end-to-end analysis and say, "Exactly where is my data in this environment?" If we're now going to start doing sub-LUN tiering and all sorts of other things, how can I visualize what the impacts are on my host and my guests?

Rick: I guess I'd have to lean on the storage platform. That's where the connection between the storage platform and the virtualization platform is going to really deliver that best. I think that visibility, if they're so integrated from the command set, then they need to be integrated from a managing and monitoring tool as well.

Doug: I think there's a slight disconnect in some parts of that as well, isn't there, between the storage and VMware in terms of being able to monitor that kind of movement. Obviously, the storage only sees the box. Hopefully in the future things go on and we understand a bit better what constructs a VM, what constructs a VMDK and things like that, and maybe that'll help in the future.

Chris: I think it's interesting because people think that as we move to newer technology and something a bit more advanced, like virtualization, they think that their jobs won't be as important or as interesting. I think that the few things we've just thrown around there show you that it still will be because there's still a huge amount to understand in these environments. They're going to be as complex as ever. In fact, in some respects, they're going to be more complex than they are today.

Rick: I think that you're exactly right. What has to happen is you have to go from anybody can set up and deploy this stuff, even a five-year-old on an iPad, but you have to be the one that's going to ensure it's going to run correctly. That's where everybody is going to be pushed to the next level, and if they can do that, that's going to be the value add. You're right. It is more complex. The bar is raised, if you will, for the storage virtualization data center administrator. It's going to be tougher in your scale point, Chris, because the storage administrator is not the virtualization administrator and the reverse is the same. There might not be permissions down to disk from the server admins in the larger environments, yet the tools are capable of it. So the visibility may be removed, but the command sets are still going to go through in terms of VAAI stuff. There are a lot of scale questions that go around with these uber features around storage.

The other thing I wanted to talk about related to storage, I kind of asked Doug if he'd worked with Snapshots, was hardware assisted locking which is one of the features because Snapshots will improve with that. Again, I've not seen it, but it's one of those features that are reported to improve the VAAI. And you're right Chris, Foskett's blog especially outlined which SCSI command is available. That's really important, because VAAI, don't want to call it Version One, but the current iteration, I think it's up to five commands, doesn't necessarily mean that every storage product supports every one of these commands. That's an important takeaway.

Chris: Yeah, absolutely.

Rick: Have you guys ever worked with somebody who has gone through an upgrade of those storage processors, or Doug, in your case, was it a new product that was ready to go?

Doug: It was a brand new install, so we didn't have any hassles with upgrades. I was real fortunate.

Chris: I would imagine that people will be cautious about using new features, such as the VAAI stuff, on an existing array. I think if they've got an array that's got lots of hosts on there and they may not all be virtual hosts, they could be physical hosts. If it's a particularly large array, I think they may be very cautious with that. In some respects, if you can afford to, it would almost make sense to deploy a new piece of kit and to understand what the data profile and the performance profile is with those features turned on before you rolled it across the remainder of your environment.

Rick: "Piece of kit," I love your correct British English. For American English, piece of kit is just a new storage infrastructure? Is that what you mean?

Doug: Piece of hardware, yes. I didn't even realize I'd said that.

Rick: Ever since you told me about tat, I've started paying attention. You're right. It might make sense. Of course, definitely the virtualization-centric answer is to consolidate this purpose, this storage, this virtualization platform all in. But as you know, Chris and Doug, the enterprise is a mix. There are going to be physical servers. There are going to be non . . . a good example is AIX systems; maybe if they're VIO virtualization type of systems, they could be used in the same SAN that the VMware environment is using or mainframe LUNs or something like that. There are a lot of stakes in here, and these other platforms may not be subject to these features, and so there might be a new bottleneck to look out for.

Doug: It's funny you mentioned that actually because a friend of mine has got in their place, which is VAAI-capable, but they are a bit wary about enabling it. As a virtualization guy, I'm going, "I just don't know about this thing you put me through. No problem at all." But in his environment, he's thinking, "Hmm, I've got another brand new kit coming in. So we'll test it on that first." So that's exactly what you said is exactly what my friend is saying.

Chris: It sort of makes senses, because I can say it could potentially be a very different data profile and you really want to understand that because that may affect your design of those boxes you deploy in the future. You may want to design them slightly differently because the data profile turns out to be different.

Rick: I think the takeaway also is that an organization needs to have a representative test and development environment. If you have a production environment, that's storage processor A, fiber channel, multi-path, lots of different drive tiers, it doesn't really do you much help if your dev environment is an open file you're running on a three to five-year-old server with a bunch of ultra 320 SCSI drives over one gig iSCSCI. You see the disconnect? If your development is there and your production is way up here, you can't get that data profile. I was lucky in a couple different environments that I worked in where our storage processors were the same in dev. Maybe half the cache or we didn't put the eight gig fiber channel in dev, that type of stuff. But other than that, it was pretty representative. I think you only get that when you have formal cut offs between dev and prod. In fact, I just got into a little Twitter spat with somebody this last week. Not a Twitter spat, but just kind of an "agree to disagree" type of thing.

Chris: It's a twit piss.

Rick: There we go. Thank you. I just got into one of those with a person who said, "Just throw it all in there and leverage DRS." I kind of knew that would be their answer for CPU and memory and presumably to the future with disk, but most organizations need separation. They want that line drawn between dev and prod and all that. I think you really just have to pony up the money for those different trust zones or security zones or whatever you want to call them, workload zones, just so you can get that data profile in an environment where you can take that risk and then with confidence take it to the other environment to go all in.

Chris: I don't think anybody at this stage trusts any piece of hardware to say, "I can run all my production and all my development and all my tests on that same box and have full security, full separate tenancy for each of those different layers and be able to implement security and change control across that." You have to compromise somewhere with that level of risk. However, Rick, your touching on that locking mechanism made me think of another potential opportunity that is not virtualization related but that could be useful. That's to allow us to do long distance replication or movement of data without needing to do masses of transfers all in one hit when you do a fail over.

Rick: Like a hot block slow replicate? Or a hot block quick replicate, slow block replicate later, that type of thing?

Chris: If you think today most of the storage arrays that are around rely on replicating the entire LUN. So it's a lot of nothing. If you were using a physical server that may not be [inaudible 21:13] because they might only be small 10 gig, 5 gig, 30 gig. But when you start creating 500 gig terabyte LUNs, failing over that LUN or failing it back, that replication process is too big. It's not granular enough for the way you might want to move VMs around. Obviously, the answer would be to move it to another VMFS, and then that one might be replicated in a different direction or something like that. Imagine if you could go to the sub-LUN and you could lock out an individual block and replicate that block, then you would be able to replicate that in either direction because you could guarantee you could block it out properly.

Rick: And you could manage snaps. If you needed the whole thing, you could snap it and replicate the source and roll in the changes.

Chris: Exactly. So you can block the blocks that relate just to that VM.

Doug: The only thing I've heard in that relation, Chris, is something about SRM was at the moment I get an SRM there's LUN level, and you fail over it in the LUN level. The only thing I've heard about that is about integration with VMware and storage for that scenario is for the storage to understand what makes up the VM and what [inaudible :27] VM, so you're only replicating the VM. When you mentioned that, that's what it reminds me of is some of this stuff that's been talked about in terms of SRM and trying to break down the LUN level [inaudible 22:40] construct to the VM level.

Chris: I don't know what the next iteration of the VAAI protocols will be, but one nice thing would be to be able to do some sub-LUN replication. If you look at the way that replication occurs, it's done on some sort of device, it may be a track or some other sort of subpart of that LUN that relates to the physical hardware. There's no reason why it couldn't be done, and that would be an absolutely fantastic feature for it to allow you to replicate parts of the LUN backwards and forwards

Rick: Sure. Then it gets complicated if you're talking sub-LUN, if you're talking like an individual VM. A lot of people go just the individual LUN replication route. A cheap plug for Veeam, Veeam backup and replication will replicate individual VMs. If all scenarios are right, we can even do change block tracking on that. It can be smart, but I think for the non-virtual world, there's still a need for these solutions and just flat LUN replication may not be smart enough.

Chris: That's true.

Rick: Let's start wrapping it up here. Every podcast has its gimmick and, Chris, you're on the hot seat for this one.

Chris: Hey, excellent.

Rick: I don't know if you read the notes I sent you, but you're getting these questions. I have this segment of questions, Chris, called '"Three Views from You," and it's a view into your past, a view into your present, and what you see in the future. So Chris, in your past, in your technology experience, be it work or otherwise, what's the most interesting technical role, story or thing that you've done at some point that you can share with us?

Chris: That is a tough question. Looking back at it, I would say probably the most interesting role was a company I set up about ten years ago to sell music. Now you might say, "What's that got to do with technology?" But we basically sold digital music on the Internet before Apple did the iTunes Store. We allowed people to cut CDs themselves and we shipped them out to through the post and so on. I did all of the technical build for that. So I built the system that would allow us to cut CDs en masse, digitized all the music, and built an entire order entry system and the web system to drive that. We were thinking on our feet every day. We were coming up with new ideas. A lot of the technology didn't exist. We had to be creative, and it was just a fantastic time.

Rick: That's awesome, dude. I had no clue.

Chris: I'm sure I must have mentioned that to you before. I must have bored you with that story before.

Rick: Maybe as you were running up to the karaoke mic, I don't know. That's great, Chris. Cool. How about today? What's going on right now that you're most interested in and most passionate within your technology practice and in general?

Chris: I guess I'm really enjoying looking at the way that the virtualization side of things is going because I can see where that will trend towards. Obviously, we've abstracted what devices that we already know today, like servers, but potentially in the future that could change. That virtualization side I find is very interesting. One other thing that I've become more interested in over the years, which I guess is probably a maturity and age thing, is process around that. Rather than just looking at the pure tech, it's how we actually make something work and actually get the processes right that surround it. Of course, that's now becoming the big thing that everybody is talking about in terms of cloud, taking what is IT but actually putting it in a proper service framework allowing you to deliver that on demand.

Rick: Absolutely. I agree. The tech almost comes easy at that point.

Chris: I've walked around a million and one data centers, and we did some last year. You're looking at the box, and the box looks like the box from the other vendor and they're all the same. I find that less interesting than I ever used to. I think we've argued that one before. I don't care what cover it has on. What I'm more interested in what we can do with it.

Rick: Rock on. Looking into the future, Chris, what do you see as most interesting or exciting technology wise coming down the pike?

Chris: I sort of touched on it there a second ago in terms of the virtualization side of things, and I think that the way the cloud's going is giving us a view of that. It's really difficult to see where we're going in the future. In fact, I had a conversation with my wife last night. We had our first mobile phone in 1994. That was about the time. It was absolutely archaic. It literally did make phone calls. You look at what I have now, a multi-function device that can do so many things. I think the future is going to be more of that ability to get data anywhere, access to data, the ability to place your applications anywhere, and we'll be less bothered about the infrastructure because that will just be sorted and we'll just be putting our application wherever it is. I think that will be a completely different paradigm, and it'll be very difficult for people to get their heads around, but it will be a massive leap forward.

Rick: I agree 100%. It's one of those things that it's inevitable, and you can't deny the scale of what's going to happen in the future. Back to the phone, I'm with you. The phone I wear around my neck, which just made that beep, that thing is more capable than the first network server I used to work on, disk storage included. It just blows me away.

Chris: Exactly. You can't imagine where the technology will be because of the way it's changed even in the last 15 or 20 years. It's just unbelievable.

Rick: It's awesome. Well, hey guys. We're going to wrap it up here. If any of the listeners have comments on the podcast, you can send an email to podcast@veeam.com. I tweet on behalf of Veeam @Veeam. I'm on Twitter at RickVanover, and Chris is on Twitter at ChrisMEvans, and is it TheStorageArchitect.com, Chris?

Chris: That's right. Thank you very much for that plug.

Rick: No problem. And Doug is on Twitter at Douglas_Carson. Chris, Doug, thanks for being on the show guys.

Chris: Thanks, Rick.

Doug: Thanks a lot.

Rick: All right. Cheers.