SES clock skew detected, custom /etc/ntp.conf wanted

This document (7023094) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 5 (SES)
SUSE Linux Enterprise Server 12 Service Pack 3 (SLES 12 SP3)

Situation

The ceph health command reports clock skew detected

The administrator wants to customize /etc/ntp.conf for the ceph cluster

Any changes made to the /etc/ntp.conf get overwritten when when ceph.stage.[2-4] are applied

Resolution

Make sure time is in sync, especially among the monitor nodes, to resolve the clock skew errors. A custom /etc/ntp.conf may help. To adjust using a custom /etc/ntp.conf, do the following:

1. Make a copy of the NTP salt state and the ntp.conf.j2 template files
# cp /srv/salt/ceph/time/ntp/files/ntp.conf.j2 /srv/salt/ceph/time/ntp/files/custom-ntp.conf.j2
# cp /srv/salt/ceph/time/ntp/default.sls /srv/salt/ceph/time/ntp/custom.sls

2. Modify the custom salt state file to use the custom template file
# vi /srv/salt/ceph/time/ntp/custom.sls

Change

   - source:
       - salt://ceph/time/ntp/files/ntp.conf.j2

to

   - source:
       - salt://ceph/time/ntp/files/custom-ntp.conf.j2

3. Modify the custom NTP template file to suit your needs. In this case the changes will make NTP on all servers reference the same time source, instead of the admin node. It will also add additional time source servers. Make changes needed that will keep your cluster's time in sync.

NOTE: The following is only an example and may not work in your environment.

# vi /srv/salt/ceph/time/ntp/files/custom-ntp.conf.j2

Change

server {{ salt['pillar.get']('time_server') }}  burst iburst

to

server  0.pool.ntp.org  burst  iburst
server  1.pool.ntp.org  burst  iburst
server  2.pool.ntp.org  burst  iburst
server  3.pool.ntp.org  burst  iburst

4. Change the local global.yml file to reference your custom salt state
# vi /srv/pillar/ceph/stack/global.yml

time_ntp: custom

5. Run ceph.stage.2 to verify your changes
# salt-run state.orch ceph.stage.2
# salt '*' pillar.get time_ntp
ses5node108:
    custom
ses5node107:
    custom
<snip/>

If the time_ntp value in global.yml, in this case "custom," shows up on all nodes, then the changes have been correctly made. If not, correct any mistakes made in the previous three steps and try again.

6. Apply your changes to one node to confirm everything is configured as expected. In this case, the changes were tested on the first monitor node.
# salt 'mon*1' state.apply ceph.time
# cat /etc/ntp.conf

Take a look at /etc/ntp.conf and verify that your changes are there. If the changes are correct, then apply the changes to all nodes.

7. Run ceph.stage.3 to apply your changes to all nodes
# salt-run state.orch ceph.stage.3

8. Restart the ntpd daemon on all affected nodes
# salt '*' cmd.run 'systemctl restart ntpd.service'

9. Check that NTP is in sync on all nodes and that each server reflects the changes made.
# salt '*' cmd.run 'ntpq -p'

Cause

The default SES behavior will configure all nodes to use the admin node as the NTP time source. The admin node will use it's own time source. By using /srv/pillar/ceph/stack/global.yml, DeepSea updates will only update our shipping files. The variable will still be enforced to use the custom setup.

The cluster is primarily concerned about time being in sync among the monitor nodes and only allows a time drift of 0.5 seconds among them before detecting a clock skew. Fix time sync among the monitor nodes to remove the error.

Additional Information

Time synchronization services NTP performs, may be replaced with other services. If you want to configure SES to always use NTP, add time_init to the global.yml file.
# vi /srv/pillar/ceph/stack/global.yml

time_ntp: custom
time_init: ntp

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:7023094
  • Creation Date: 13-Jun-2018
  • Modified Date:03-Mar-2020
    • SUSE Enterprise Storage
    • SUSE Linux Enterprise Server

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center