Virtualization

10 recommendations for working in a VMware ESX environment

Day to day management of virtual machines yields some interesting lessons. Here are 10 recommendations for working with VMware vCenter 5.1.

VMware_logo_wht_RGB_300dpi_081613.jpg
Virtualization brings possibilities that were previously unheard of, or at least more difficult to achieve with physical systems. Most people don't equate IT with creativity (at least outside the field), but I have found that working with virtual systems has presented all sorts of creative scenarios. The ability to easily duplicate virtual machines; to take snapshots to save the current system status and "roll back" to this point in time later; to spin up and bring down test systems within minutes - all this has freed up IT staff for bigger and better things. Like many time-saving developments in technology I have found virtualization hasn't reduced headcount but actually increased it since it opens the door to more options.

I've worked with VMware environments for a few years now and currently spend a lot of hands-on time with 5.1. Here are some tips I've put together based on my day-to-day experiences, using the VMware vSphere client for Windows.

1. Be careful when cloning Windows systems on a domain

We have several virtual machine templates for Windows XP, 7 and 2008. At present, I leave them all off the domain, keeping them in the standard WORKGROUP instead. We join them to the domain after cloning them, because weird things can happen when you clone operating systems joined to an Active Directory domain. The clone or the original domain member might experience logon problems since now there is an imposter afoot. The issue is caused by the fact that each Windows computer has a unique ID - or to be precise, a SID (Security Identifier). This is like a fingerprint, and it is different for all Windows computers, or rather it should be for things to work properly. Mark Russinovich wrote a good article about this topic back in 2009, pointing out that while there were some misconceptions about how the SID impacts cloned machines, it is still recommended to use Microsoft Sysprep to smooth out this process (for domain or non-domain systems.) 

Since many of the systems I clone are for test purposes only, I don't always use Sysprep, but I did so on a domain server recently and that managed the transition quite nicely. Here are some instructions for XP, Windows 7 and Windows 2008.

2. Converting Windows XP physical systems to VMs might produce network driver problems

Microsoft is ending support for Windows XP on April 8, 2014, so hopefully you're planning to retire this OS from your environment by then, at least on critical or production systems. However, in the meantime if you find yourself cloning a physical XP system into a virtual machine and you just can't get the default network drivers to work, VMWare has a solution, which involves downloading and using the E1000 network driver in your virtual machine. This has solved a couple of frustrating issues that didn't seem to impact freshly built VMs.

3. Shut down source virtual machines before cloning, if possible

VMware allows you to clone a running operating system without impact; this process is known as "hot cloning". However, I've seen it take quite a long time on some systems, particularly those with a lot of files. If possible, try shutting the system down (cold cloning) to speed things up. You'll lose the benefit of keeping the OS running throughout the process, but if it's not a critical machine, the tradeoff can be worth it, particularly if you're in a hurry. Another advantage is that you won't have to worry about data changing on the source and not being reflected on the target. If this is likely, you should definitely shut the VM down first if the data is essential.

4. You can clone VMs simultaneously, but be wary of performance issues

A great element of VMware is the ability to clone a single source virtual machine to multiple new VMs at once. I've done this to five or six new VMs at a time. However, this can result in a performance drain upon your VMware ESX hosts and the underlying VMs. I've even seen VMs on other ESX hosts start to sputter during a batch cloning job.  

Each VMware environment is different, of course, so if you have sufficient CPU, memory and network resources this may not be a factor, but be aware it might crop up and prepare to address it by canceling cloning operations or reallocating resources if possible.

5. Balance target VMs among data stores

If you have multiple data stores in your ESX environment, don't load up the first one with VMs then gradually use the remaining ones as needed in a "fill then spill" process. Try spreading the virtual hosts out among your data stores so each one isn't overloaded. You want to make the most of the resources provided by your ESX hosts/data stores to get them up to a good capacity level, but one shouldn't be top-heavier than another if you can avoid it. VMware offers a product called Storage DRS which can provide intelligent load balancing to take this concept even further.

6. Keep an eye on alerts

The VMware vSphere client lets you view your ESX hosts and check for any alarms such as resource or host problems:

Figure A

vsphere_alert_A.png

Better yet, you can configure these alarms to provide you with direct notification when issues occur. Click on your main VCenter system, then choose the Alarms tab and click Definitions (to the right of View):

Figure B

vsphere_alert_B.png

You can open each alarm to view details and set the desired configuration:

Figure C

vsphere_alert_C.png

Click the Triggers tab to set the conditions you want to put in place:

Figure D

vsphere_alert_D.png

In this example, I want to set the Host CPU Usage alarm to define a warning if the percentage is at 75 or above for 5 minutes, and to send an alert if the percentage is at 90 or above for the same time.

I can specify Reporting conditions to repeat the alarm if the percentage changes, as well as whether the alarm should repeat:

Figure E

vsphere_alert_E.png

Finally, I can specify the appropriate action to take - in this case to send a notification email:

Figure F

vsphere_alert_F.png

In the above screenshot I set the email to go out if the condition changes from warning (yellow exclamation mark) to alert (red exclamation mark). You can set alerts if the condition changes from normal to warning, warning to alert, alert back down to warning, or warning back down to normal.

You can also add alarms to a virtual host by right-clicking it, choosing Alarm then Add Alarm (or clicking the VM and pressing Ctrl-M). You can then administer these individual alarms using the Alarm tab for that host.

I highly recommend you set disk storage alarms on your data stores (as well as other critical elements). Few things are scarier than running out of disk space in a virtual environment where multiple machines might go down in that scenario.

7. Get an in-depth view of your VM hosts

It's handy to view the sizes of your virtual machines to keep track of what storage space is being used or to see which virtual machines are on which data store. Press Ctrl-Shift-D to bring up the "Datastores and Datastore Clusters" view in the vSphere client:

Figure G

vspheredatastore_G.png

Click the Virtual Machines tab and you can get a close-up view of the virtual systems and their status:

Figure H

vspheredatastore_H.png

8. Take advantage of built-in performance counters

I'll admit I was never a fan of the performance counters available in Windows systems. They seem too kludgy and cumbersome to set up. VMware offers some performance details under the Performance tab of each VM, which can give you a quick and easy glimpse as to how the virtual host is running:

Figure I

vmperformance_I.png

Figure J

vmperformance_J.png

9. Use snapshots… but remove them later

Snapshots make virtualization fun, since you can restore a host to a prior state after you've completely mangled it (hopefully through experimentation only on a test host, but you get the picture). Snapshots aren't without a price tag, however – they take up disk space and slow down the cloning process, so they should be periodically removed once it is unnecessary to keep them.

There are scripts that can monitor snapshotsas well as remove them, but make sure to test these first and don't run the latter unless you're positive the snapshots won't be needed.

10. Know your database

Your VMware ESX environment has a database, whether running on Oracle, a SQL instance or even IBM DB2. If that database has problems, so will your virtual machines. If you're lucky the VMs might still remain running in the event of a database failure, but I've seen scenarios where some virtual hosts went down hard – and they couldn't be brought back up right away since the database had problems.

Get to know how your database works, where it resides, how to support it, and all other relevant details in the event of a problem with it. Many system and network administrators have strong roots in one area and weaker experience in another. If databases aren't your thing, learn all you can about the one sitting underneath your VMware systems so that if (when) the day comes when you have to support it, you won't be stumbling blindly through Google searches. I've seen issues whereby a SQL Server Express database used for VMware reached the 4 Gb limit then shut down. VMware provides a resolution for this, but it's temporary. I have even seen a problem whereby a security scan against a database server tanked the VMware database and rendered it unable to mount due to lingering file locks. Don't wait for zero-hour to be ready for such an incident.

Conclusion

These tips are just a small drop in the bucket of what you can learn as a VMware administrator, and since the details are always changing I have found that navigating the menus, learning the layout of the environment, and getting familiar with the available functions has been the best tip of all. VMware allows you multiple ways to perform similar tasks, such as copying virtual image folders from the command line or even scripting the automated deployment of VM clones, something I am about to start researching.  

The rewards of virtualization can be as strong as the challenges involved. It's worth noting that while you can create new systems much more rapidly than before, careless mistakes can have louder and more harmful repercussions. For instance, few if any admins would accidentally delete a physical domain controller (unless they were really confused), but nowadays you can do so in seconds if you're not constantly on your guard and paying close attention. Check, double-check, and triple-check your actions if need be before you make changes to your VMware environment – it's one place where I truly don't mind those otherwise annoying Windows prompts asking, "Are you sure you want to proceed?"

About

Scott Matteson is a senior systems administrator and freelance technical writer who also performs consulting work for small organizations. He resides in the Greater Boston area with his wife and three children.

Editor's Picks