Unrecoverable NTP time drift in XEN HVM domU if clocksource=tsc after live migration

The document is only relevant if  clocksource=tsc is used.

Only older hardware is affected, that does not yet support TSC Scaling.

The solution is to use chrony because it is more resilient when it comes to CPU clockspeed frequency differences and recovers better from time drifts. 

During the boot of the host, XEN will estimate the TSC frequency and report it in xl dmesg:  

# xl dmesg | grep Hz
  (XEN) Platform timer is 14.318MHz HPET
  (XEN) Detected 2399.999 MHz processor.

The MHz value varies across reboots and is not 100% accurate. The value is used to decide if TSC access needs to be emulated. If source and destination hosts calculated a different value, TSC emulation will be used.

SUSE introduced the parameter suse_vtsc_tolerance=KHz to allow for some variance between CPU clockspeed frequencies when migrating between hardware. It relaxes this check to provide native, unemulated access to the RTC.

If the value is too large, live migration will take place but might also cause a NTP drift within a HVM domU that is irrecoverable. ntpd can only handle differences up to 500ppm.
1ppm is one microsecond difference per second.
500ppm equates roughly to ~1MHz at a CPU base frequency of 2GHz.
If 500ppm is exceeded ntpd silently stops working.

During the boot of the dom0 the kernel will also estimate the TSC frequency and report it in dmesg:

# dmesg | grep -Ei '(clock|tsc|hz)'
...
[    0.012021] tsc: Detected 2399.998 MHz processor
...

A NTP client can run in dom0 to compare the speed of the system clock with the speed of an external reference clock. The calculated drift will be used to adjust the advancing of time. Both ntpd and chronyd write the drift to a file at regular intervals. Extra logging can be enabled to see how the drift varies over time (ntpd writes statistics to "loopstats", chronyd writes them to "tracking.log").

The real TSC frequency can be estimated from the reported MHz value and the drift value.

Similar to dom0, a HVM domU will also estimate the TSC frequency during boot time. The assumed frequency will be used for the entire runtime of the domU. In case the clocksource is forced to be 'tsc' instead of 'xen' (via cmdline option "clocksource=tsc nohz=off highres=off"), the actual hardware frequency of the TSC is expected to remain the same.

Based on the reported MHz values and the reported drift on both dom0s, the drift within domU after the live migration can be estimated. The domU kernel expects the TSC to advance in a certain speed. If the frequency differs on the other host, time will run faster if the frequency is lower, and it will run slower if the frequency is higher. A NTP client can correct such differences up to a certain point. chronyd can handle a significant higher drift than ntpd.

The drift is in ppm.

src_dom0_real_Hz   ?dst_dom0_real_Hz?      (us+drift)*src_dom0_real_Hz
---------------- = ------------------- === ---------------------------
      us               us+drift                       us

Maximal target frequency for migration from a 2500 MHz host:
(((1000*1000)+500)* (2.5*1000*1000*1000)) / (1000*1000) = 2501.250 MHz

To enable tracking in ntpd, add this to ntp.conf:
  statsdir /var/lib/ntp/drift/
  filegen loopstats  file loopstats  type day enable

In case apparmor is enabled, ntpd lacks permissions to write stats

To enable tracking in chrondy, add this to chrony.conf:
  logdir /var/log/chrony
  log tracking

Restart the daemons.

The statistic files will be updated in varying intervals.