Time drifting when running a Linux guest under VMware ESX server

This document (3858673) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 11
SUSE Linux Enterprise Server 10
VMware ESX server
 

Situation

Time drifting when running a Linux guest under VMware ESX server.
When running eDirectory on the guest, ndsrepair -T reports that time is not in sync.
ERROR: -659 adding a server to the tree
 
 
BACKGOUND:
An operating system needs to have some way to schedule work in the future at a precise time. A processor supporting such an operating system must have a programmable interrupt timer that can periodically interrupt the processor in order for it to run scheduled work. This periodic interrupt is known as a system clock tick. All activities are scheduled using this constant "heartbeat". Within the Linux kernel time is seen as the amount of clock ticks since the system booted. This measurement is known as jiffies.
 
The Linux 2.6 kernel in SLES9 changes the base interrupt rate as compared to the 2.4 kernel in SLES8 from 100 per second to 1000 per second. (SLES 10 SP1's kernel is set to 250 HZ.) A dual-processor Linux 2.6 kernel can fire up to 3000 per second. This is usually not an issue when running on a bare metal server under moderate load. However, issues can easily arise in a virtualized environment where multiple instances of the Linux kernel are in competition for the host's actual hardware based processor's attention. When the system gets busy for extended periods lost ticks can occur (in a virtualized environment little actual guest OS activity is required). These lost ticks are usually due to I/O and system services that must wake up and mask or disable interrupts for a time while they run. When the interrupts are disabled for an extended amount of time ticks will pile up and the kernel can lose track of their count - they are consideredlost.
 
Due to the descrepency of how many interrupts the Linux kernel guest occurances can generate and the amount the ESX host kernel can service, lost ticks are a frequent concern. Code was added to the 2.6 kernel to detect lost ticks and correct for them. However, it is difficult at times to accurately detect when a tick was lost. Due to the way x86 queues the interrupt handlers an extra tick may be added by this code when, in fact, none were actually lost. These extra ticks can lead to time gaining or moving forward beyond the server's time source. Lost ticks in themselves lead to time being lost (time slowing from the source).

Resolution

To minimize time advancing beyond the time source due to the kernel's compensation code adding too many ticks use the following boot parameter:
clock=pit
(For a SLES 10 SP1 kernel running under ESX 3.x this is likely the only setting that will have to be changed.)
The following settings and recommendations for Linux guests running under ESX should avoid timesync issues in this environment.
 
NOTE: SLES 11 requires no kernel parameters.

SETUP NTP ON THE VIRTUALIZED GUESTS

a. Ensure that an external reliable NTP time source is used both for the ESX host as well as its virtualized guests. The timesource should not be the ESX host itself nor a virtual server running under it. Doing otherwise will only add to a virtual machine's interrupt load and exacerbate the issue.
b. Set up the NTP Source.
  • Modify the /etc/ntp.conf file. Comment out the following entries so the local clock will not be used by placing # signs before them:
    #server 127.127.1.0 # local clock
    #fudge 127.127.1.0 stratum 10
  • Add an entry for this time source at the end of the file replacing x.x.x.x with the server's IP address:
    server x.x.x.x prefer
c. Slam time, start the NTP daemon and verify time.
  • Stop the NTP daemon. On SLES10:
    /etc/init.d/ntp stop
    On SLES9:
    /etc/init.d/xntpd stop
  • Slam the current time by typing:
    ntpdate x.x.x.x
    (where x.x.x.x is the NTP source's IP address)
    (Force this command several times until the offset displayed is less than one second.)
  • Start the NTP daemon again. On SLES10:
    /etc/init.d/ntp start
    On SLES9:
    /etc/init.d/xntpd
  • Wait for this server to show a reach of 377 by typing:
    ntpq -p
    (This may take 15-20 minutes.)
 
Make changes to the VMware Environment
ESX 2.x only -Misc.TimerHardPeriod
This helps with time lagging. If time is lagging on a ESX 3.x server with SLES 10 SP1 kernels it is likely NTP is misconfigured, the source is unreliable or there is too much load on the ESX server. Moving virtual machines off the server to another and\or tuning the priority of the guest may help as this gives the guests more processor cycles. VMware recommends not changing the default values in ESX 3.0 for the max and min times.
Using the ESX 2.0 Status Monitor select Options - Advanced Settings - Scroll to the setting"Misc.TimerHardPeriod". Change the value from the default of 1000 to 333 (or 250) and OK it. This change is dynamic and takes place immediately.
 
 
 
Change the Kernel Boot Parameters
ESX 2.x and 3.x - clock=pit

This change helps with overcorrection of time - time moving forward due to the the extra ticks generated by the kernel's compensation code.
Within the guest, open the /boot/grub/menu.lst file within vi and add "clock=pit" to the end of the kernel line in the title Linux section.
NOTE: There was a report of clock=pit causing a reboot when clustering services start when running in a clustering environment. The recommendation was to change this to clock=pmtmr.
Below is an example:
title Linux
kernel (hd0,2)/boot/vmlinuz root=/dev/hda3 vga=0x317 selinux=0 splash=silent
resume=/dev/hda1 elevator=cfq showopts clock=pit
If using LILO, add the clock=pit parameter to theappend=line in the/etc/lilo.conffile as follows:
append="resume=/dev/hda6 splash=silent clock=pit"
Remember to run /sbin/lilo after editing lilo.conf, so that the edit takes effect.

Whether using LILO or GRUB, please reboot the server after adding this parameter.

Normally the above two VMware ESX server and guest changes should be enough to resolve NTP time issues within the guests. If NTP time issues persist then one may perform this additional step. It is not recommended to make this change unless all other avenues have been investigated.

- Locate the *.vmx configuration file for the guest on the ESX filesystem. (Default location is /root/vmware.) Open this file in vi and change the statement "tools.syncTime=FALSE" to "tools.syncTime=TRUE". For this change to take effect, the SLES9 guest will need to be completely powered off, not just reset, in order to re-read its configuration file. When powering the guest back up the guest should notice that its configuration file has been changed and ask if it should be re-read. Answer "yes" to this question.

Should you find NTP time is not being reliably synchronized the following steps may be useful:
- Verify that the choosen time souce has given us time and is trusted:
Ensure that when typing:
ntpq -p
that there is an asterisk by the chosen time provider entry.
The virtual machine may need to its APIC and/or ACPI disabled at boot time before loading the OS in addition to these steps.
If an asterisk is not seen using the ntpq -p command APIC could be interferring with time synchronization. This can be disabled by passing the "noapic" parameter at bootup by adding this parameter to the boot switch in /boot/grub/menu.lst.

Type
ntpd
At the command window grab a list of association indexes by typing "as". Then grab a detailed debug report on this server by typing "rv assid" where assid is the number seen in the previous step. Many of the values displayed are useful in debugging NTP. One of the most useful is the flash value. This displays the results of the sanity checks placed on the incoming NTP packets. This value should be "flash=00 ok".

Additional Information

Perl Script for testing the time drift:

By using the following perl script it is possible to test the success of the suggestions mentioned above:

#!/usr/bin/perl

# Get drift
my $lines = `ntpdate -q pool.ntp.org`;
my @lines = split(/\s/,$lines);
my $dummy = pop(@lines);
my $drift = pop(@lines);

my $nowstring = `date +"%Y%m%d-%H%M%S"`;
chomp $nowstring;
print "$nowstring -> $drift\n";

Save this script e.g. into the file /usr/local/bin/timedrift-check.pl and make it executable (by using chmod +x).
The script prints a line in the following format:
->
When starting the script frequently you can see how the time drift evolve.

VMware Knowledgebase Articles
TID history

Formerly known as TID# 10100597

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:3858673
  • Creation Date: 29-Feb-2008
  • Modified Date:04-Mar-2021
    • SUSE Linux Enterprise Desktop
    • SUSE Linux Enterprise Server

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center
Report a Software Vulnerability Submit Tips, Tricks, and Tools Download Free Tools