According to VMware, admins should not designate a datastore for operating system virtual disk files. VMware vExpert Rick Vanover offers his take on this blanket recommendation.
When designing storage for VMware vSphere installations, you have to make a lot of decisions. For instance, there are a number of use cases to have a designated datastore for operating systems. The most frequent use case is to take advantage of primary storage deduplication; this is usually the case with NetApp storage, which guarantees at least a 50% deduplication benefit for virtual machines. Other benefits of this practice include simple organization and basic tiering of the virtual machines. For example, the operating system drive letter (C:\) isn't usually as critical as a database volume, so it may make sense to separate the two and put the database on a higher tier of storage.
However, a recent VMware blog post makes a blanket recommendation to not designate a datastore for operating system virtual disk files. VMware outlines a number of issues with this practice, including load spikes, risk of hypervisor swapping, and snapshots filling up a volume and causing increased contention.
I think it's hard to nail down a specific configuration in this category -- too many factors come into play (such as questions about the workload requirements and what type of storage is available) to make blanket recommendations.
I also feel that VMware is somewhat overlooking thin-provisioned LUNs on the storage processor. In regards to space management, a thin-provisioned LUN (and therefore datastore) is a wonderful convenience. The thin-provisioned LUN allows a datastore to exist with comfortable amounts of free space in vSphere, though some of the other issues identified in the VMware post may still apply. (Managing aggregated oversubscription is an entirely separate discussion.)
My virtualization practice has been to tier by the overall virtual machine's workload, with the occasional split of the operating system and data virtual disks to a higher tier. In most situations, I keep them together.
Weigh in with your thoughts about this contentious topic.