PCs

Take a stance on virtual machine time sync

The seemingly inevitable trend towards virtualization brings one important point to Windows administrators: time settings. Make a decision on how to approach VM time early on before it bites you.

As Windows administrators seem to have an increasing percentage of virtual machines, the concept of system time needs detailed focus. This becomes more important as the virtual systems contain more and more elements of the core workload.

A virtual machine can get its time from the host system, which in many situations is convenient and a good enough solution. Windows virtual machines that are members of a domain are also instructed to get their time from the domain. As we can see where this is going, it is important to determine one authoritative time source for virtual machines. I prefer to have the domain manage the time and have the virtual machine not sync to the host. This configuration can make multiple time zone configurations on the same virtualization host a little more transparent.

For VMware platforms, the VMware Tools Properties dialog box has an option to synchronize the time with the host to the guest. Figure A shows this option in VMware tools. Figure A

Figure A

Current versions of VMware tools do not enable this configuration by default, but the option is available and may be configured for older virtual machines. Other virtualization platforms, such as Sun xVM VirtualBox with the Guest Additions installation, have a default option for time being synchronized to the host, and it is less than intuitive to change that configuration. This becomes extremely important if a domain controller is run as a virtual machine. A common practice is to not synchronize the time to the host system, but to synchronize it to a trusted NTP time server.

Some advance planning in regards to the time configuration of a virtual environment will prevent surprises as environments grow.

Stay on top of the latest Windows Server 2003 and Windows Server 2008 tips and tricks with our free Windows Server newsletter, delivered each Wednesday. Automatically sign up today!

About

Rick Vanover is a software strategy specialist for Veeam Software, based in Columbus, Ohio. Rick has years of IT experience and focuses on virtualization, Windows-based server administration, and system hardware.

12 comments
techrepublic.com.com
techrepublic.com.com

In the past I've had very bad experiences using the suggested approach. For the farms I worked with (2/3/35 ESX per farm) we chose the opposite direction. All VMs were synced to the hosts. The ESX servers ran a full blown NTP configuration, with at least 3 (internal) NTP servers as time source, which were sync peers and each had 3 upstream NTP servers on the internet. The Domain Controlers used the very same NTP infrastructure so they always agreed with the ESX hosts regarding to time. In this scenario, with heavily loaded ESX servers, we ran some linux VMs that were not synced with their hosts but were NTP clients theselves. Their NTP config was identical to that of the ESXes. They all experienced extremely drifing clocks. NTP could never get synchronised to any NTP server; the clock could float back and forth up to 10 hours within a week of run-time. As far as I remember the problem is that the VMs do not get a constant rate of CPU cycles. The effective rate depends on the load of the ESX hosts. For the VMs it has the effect that you would get when you could randomly change the clock speed of the CPU without the OS knowing about that. Every timekeeping algorithm is based on the presumption that the CPU has a fixed clock rate; a presumption that is false for VMs. Maybe you're saved by the fact that Windows uses a simplified version of NTP for it's internal timekeeping across the domain?

WayneAndersen
WayneAndersen

Does anybody know what the times settings would be for Microsoft's Hyper-V?

Kgottleib
Kgottleib

I'm sorry MR author, but you need to understand virtualization a little better. I have dealt with VM time sync issues in 3 enterprise organizations and can tell this, sync time with your host server. VMs don't have a hardware clock, they have a virtual clock that uses vCPU cycles to manage it. This means when the VM powers down the clock stops. When the VM powers up it would have to sync with a time provider on the wire if it isn't set to sync with the host. If this time provider is not available because its too busy to handle the request, a DNS issue, or whatever, you now have inaccurate time. Some apps will fail when the clock is too far out of sync with the DCs in the domain. VMware tools will sync the VM's clock every minute. this is your best option. Even if an application is running that requires windows time service to run, like a DC which is a time provider to member servers and client PCs, you still want to sync with the host server and set a value in w32time in the register to NoSync so the VM doesn't have more than one source. On that note, always make sure that your esx host has a registered time provider. you can verify this by running ntpq- p in the service console and look for the asteric, that indicates a registered time provder. Also, another useful tidbit that vmware doesn't want you to know is that there is a bug in the ntp service running in the service console that requires you to enter IP addresses rather than FQDN. This is why VMware's newest version esx 3.5 and VC 2.5 has a new option in VC to configure NTP in the GUI that requires you to enter IP addresses of the time provider you choose as opposed to FQDN.

Kgottleib
Kgottleib

VMware's ntp has a bug, can't register a time provider with FQDN, that is why your host servers drifted. You are correct about a busy host server, there are custom lines of code you can use in the .vmx that will allow better access to host CPU that will improve time keeping, I have the syntax on my thumb drive and I don't have it with me so I can't share it with you at this time, send me an email to kgottleib@gmail.com and I will share it with you. Otherwise, if time is lagging in a VM because of busy CPU, would you rather have it sync every minute or the default time frame windows provides, i think it is 4 hours. ALso, the algorythm in windows will speed up its clock if it is lagging, you don't want this to happen. VM time sync is an issue on heavily utilized servers, make sure your host server doesn't exceed 60% and you should be OK. Otherwise, consider migrating your busy apps to physical servers or add more hosts to your architecture and or consider CPU affinity settings, make sure you map your VM to x +1 cores.

ScottCopus
ScottCopus

If a VM is off for a couple hours, it's time isn't corrected? Do the VMs not do any kind of timesync during the VM BIOS POST? -Macgyver

WayneAndersen
WayneAndersen

Thanks for your post Kgottleib. What you say makes sense. If a VM host is down for a couple hours, say, all the VM's will be off when they come back on line. With an external time source it might take a while to get back in sync. In the mean time they would be running with a large time differential which is never a good thing.

dwilga
dwilga

I beg to differ. I have absolutely no issue with any of my VMWare's ESX servers keeping track of NTP time with my Windows server acting as a primary, with a fallback to (several) public NTP servers. I have collected a set of simple tweaks from various sites on the internet and my staff uses these as an ESX server "up and running guide." These tweaks never fail to get us a fully functional ESX server, including NTP client ability. You might want to try these excerpts from my file as simple tweaks to your own configs. Note: "Martin" is my local NTP server. /etc/ntp.conf ======================================== restrict kod nomodify notrap noquery nopeer restrict 127.0.0.1 server Martin server 0.us.pool.ntp.org server 1.us.pool.ntp.org server 2.us.pool.ntp.org server 3.us.pool.ntp.org driftfile /var/lib/ntp/drift Restart the NTP Client service: =============================== service ntpd restart Enable the NTP daemon to autostart: =================================== chkconfig --level 345 ntpd on Set the local hardware clock to the NTP synchronized local system time: =============================== hwclock --systohc See the offset (in seconds) between the local clock and the source clock: =============================== ntpdate -q martin

techrepublic.com.com
techrepublic.com.com

FYI: It was not our ESXes that were drifting, but the Virtual Machines running a functional NTP on Suse Linux Enterprise Server 10. We had no problems keeping the ESX farms synchronised. In our environment the load on the VMs fluctuates heavily and unpredictably, resulting in frequent short-term CPU exhaustion on the ESXen. This is a major challenge in capacity management over the VMs since manual or automatic Vmotion places a heavy load on the ESXen as well. I agree that a 60% average load would be optimal but not every shop will be able to effectively maintain this load, especially not if the VMs are running batch-oriented applications.

Kgottleib
Kgottleib

Scott, When VMs are set to sync with the ESX host server they do sync with the service console upon boot, the parameters for time sync lie in the configuration file, .vmx, which is read when the VM is started. If it is set to TRUE it will sync, if it is set to FALSE it won't. If it isn't set up to sync then you better have windows time service enabled. Like I said, if you use windows time you can run into problems, see my initial post.

Kgottleib
Kgottleib

Let me correct you, the VM host server has a hardware clock, make sure it is in sync with the service console, you can do this by running this command in your service console: hwclock --systohc also this: chkconfig - -level 345 ntpd on This will make sure Ntp starts on reboot this way if you host server is down when it comes back up your time should be good to go.

Kurse
Kurse

Please post your code snippet for the .vmx files, it could possibly aid many subscribers here on TR, and I am interested, myself.

Kgottleib
Kgottleib

Sure, I've resolved issues with both ESX drift and VM clock slow down because of excessive CPU usage on the host. There were 6 hosts in my environment that had this problem, each was running at 80% CPU because of the shitty, CPU intensive apps, each configured with 2 vCPU. I convinced the apps groups the linux VMs didn't need 2 CPUs adn they didn't because once we set them to 1 vCPU they performed the same and the CPU usage of the host was reduced to about 68%. The windows VM's clocks were still lagging, so I configured CPU affinity mapping the 2 vCPUs to 3 cores. That helped substantially and reduced CPU usage even more, to about 64% on the host. finally, I ended up getting a custom line of code for the .vmx that resolved the issue in full. Send me an email so I have your contact info and I will send it to you tomorrow. Great chatting with you on this post. Still disagree with the windows time service analysis though, I got into trouble with it and will always snyc with the host now.

Editor's Picks