Virtualization

Why VMware doesn't recommend datastore layout by OS

According to VMware, admins should not designate a datastore for operating system virtual disk files. VMware vExpert Rick Vanover offers his take on this blanket recommendation.

When designing storage for VMware vSphere installations, you have to make a lot of decisions. For instance, there are a number of use cases to have a designated datastore for operating systems. The most frequent use case is to take advantage of primary storage deduplication; this is usually the case with NetApp storage, which guarantees at least a 50% deduplication benefit for virtual machines. Other benefits of this practice include simple organization and basic tiering of the virtual machines. For example, the operating system drive letter (C:\) isn't usually as critical as a database volume, so it may make sense to separate the two and put the database on a higher tier of storage.

However, a recent VMware blog post makes a blanket recommendation to not designate a datastore for operating system virtual disk files. VMware outlines a number of issues with this practice, including load spikes, risk of hypervisor swapping, and snapshots filling up a volume and causing increased contention.

I think it's hard to nail down a specific configuration in this category -- too many factors come into play (such as questions about the workload requirements and what type of storage is available) to make blanket recommendations.

I also feel that VMware is somewhat overlooking thin-provisioned LUNs on the storage processor. In regards to space management, a thin-provisioned LUN (and therefore datastore) is a wonderful convenience. The thin-provisioned LUN allows a datastore to exist with comfortable amounts of free space in vSphere, though some of the other issues identified in the VMware post may still apply. (Managing aggregated oversubscription is an entirely separate discussion.)

My virtualization practice has been to tier by the overall virtual machine's workload, with the occasional split of the operating system and data virtual disks to a higher tier. In most situations, I keep them together.

Weigh in with your thoughts about this contentious topic.

About

Rick Vanover is a software strategy specialist for Veeam Software, based in Columbus, Ohio. Rick has years of IT experience and focuses on virtualization, Windows-based server administration, and system hardware.

7 comments
nwallette
nwallette

Doesn't the VMWare Host OS have block caching? Doesn't your SAN? So even if you decided to start 20 VMs all at the same time, the disk I/O burden would conceivably be less of a concern than the spikey load on the CPU. Generally speaking, OS disks don't tend to be the cause I/O stress during normal operation. Has anyone else found this to be the case? Even if you have your swap file on the system volume, if your swap I/O is considerable, that seems to indicate a performance issue in regards to the amount of memory allocated to the VM. As for snapshots... OK, they'll pile up in one place, and if you use thin provisioning, you could starve the available disk space for other OSes to consume. If you don't, you only starve the space other snapshots could consume. You should be monitoring disk space anyway, so if you see that happen, it's not necessarily the end of the world. Clear some snapshots, grow the LUN, or agree to give up your time capsule. Big deal. So, my self-offending blanket philosophy: Always take blanket statements with a grain of salt.

george.hickey
george.hickey

Our arrays are somewhat older so thin provisioning isn't an option on the LUN side. We could do it on vSphere but haven't bothered as we've gone a different route. We were using a small number of large data stores, each containing multiple VM's but found that we were getting strange IO errors when more than one VM in a datastore engaged in a high-IO activity at the same time - I had been told by a VMware trainer while on a course that VMware can and does lock datastores when writing. This isn't a problem if you only have one VM in a datastore, or if the VMs aren't attempting to write at exactly the same time. It does become a problem when you have two or more VMs trying to do intensive IO activities at the same time. We went down the route of assigning at least two individual datastores to every VM, one for the O/S and one for data. Some of our bigger VMs have 4 or even 5 individual datastores. Let me stress - we are a small environment with less than fifty VM's currently so this approach is manageable for us... I can't see it being workable for environments with hundreds of VMs. Just thought I'd share!

oldbaritone
oldbaritone

Windows was originally built on DOS, and has never had good housekeeping for the multi-user environment. It has gotten somewhat better over the years, but there are still nagging throwbacks that "bite you" in the virtualization arena. Come on, if you're going to run 10 virtual machines with exactly the same OS on the same physical machine, the OS operational programs should be able to use one common copy of the code, with separate registration and operation data for each machine - i.e. one tree that lives as "C:\WINDOWS" (or at least "C:\WINDOWS\SYSTEM") for all 10 machines. It's the same runtime, and the same install disk! And that way, updates applied once would reflect on all 10 machines. But to paraphrase a grade-school evaluation - Windows "does not work and play well with others" - it never has, and it still doesn't. If you try to create a shared datastore for the OS files, you're creating a lot of headaches for yourself. So instead, you create 10 copies of the OS, 9 of which should be unnecessary, and make 10 updates every month, 9 of which should be unnecessary. And the physical machine spends time swapping 10 copies of the same files, 9 of which should be unnecessary, creating substantial unnecessary system overhead.

BenMitchell
BenMitchell

With thin provisioning there is very little reason to split off the OS from the Data onto different datastores. Ben Mitchell www.complete-geek.com

Albert Widjaja
Albert Widjaja

George, I'm on a SME now with around 90-100 VMs at the moment mostly application servers with a few DB server as VM, so far the workload for the iSCSI SAN is OK eventhough I put one VM of both disk (C: and Data) into the same datastore. Even in my scenario, all ofthe VM deployed only have one VMDK 80 GB and then I partition that alter on manually to become another Data partition. When there is Storage contention problem, I can just migrate the VM into another datastore with lesser VM.

nwallette
nwallette

No offense sir.. but that wouldn't work in real life. The redundant cruft is necessary to treat an OS and its applications like an atomic unit. The one exception I can think of: Solaris gives you some options with Zones.. you can choose to duplicate the OS environment for every separate container, or you can share one. This gives you the choice of updating everything all at once, or requiring cascaded updates (root zone, then each child zone.) But it still won't let you share with other installations. And I wouldn't recommend running an updated OS long before rebooting to bring those changes into a state of parity again. The key here is that an OS (and its applications!) must be aware of changes made to libraries. Otherwise, things end up inconsistent and the code that should be available via a known symbol is this particular file, no longer is. Then what? Crash. Deduplication on the datastore helps with this by only needing to store one physical copy of every unique arbitrary block of data. So even though there is duplication in the guest OS filesystems, that duplication is eliminated in the host OS filesystem.