How to monitor NTP on Linux nodes with Cluster Monitoring in Rancher v2.2.x+

This document (000020118) is provided subject to the disclaimer at the end of this document.

Situation

Task

Time drift between nodes in a Kubernetes cluster can create a range of issues, from a difficulty to correlate application log message timestamps across nodes, to a loss of etcd quorum (given the time sensitive nature of the consensus algorithm used in etcd).

Using Rancher, you can monitor the state and processes of your cluster nodes, Kubernetes components, and software deployments through integration with Prometheus, a leading open-source monitoring solution.

This article details how to monitor time drift, via the Network Time Protocol (NTP), on Linux nodes within Rancher Kubernetes Engine (RKE) or Rancher v2.x provisioned clusters.

Pre-requisites

  • A Rancher v2.x instance, starting at v2.2.0 and above
  • A Rancher Kubernetes Engine (RKE) CLI or Rancher v2.x provisioned Kubernetes cluster with Cluster Monitoring enabled, with Monitoring Version 0.2.0+
  • ntp configured on Linux nodes in the cluster (refer to the documentation for your Linux distribution on enabling and configuring ntp)

Steps

Enable the NTP collector on the Node Exporter DaemonSet
  1. Within the Rancher UI cluster view for the relevant cluster, navigate to Tools -> Monitoring
  2. In the bottom-right corner of the form, click Show advanced options
  3. Click Add Answer
  4. Configure the variable exporter-node.collectors.ntp.enabled with value true
  5. Click Save
Configure an alert for NTP time drift
  1. Within the Rancher UI cluster view for the relevant cluster, navigate to Tools -> Alerts
  2. On the A set of alerts for node Alert Group click Add Alert Rule
  3. Set Name to Node NTP time drift equal to or greater than 1 second
  4. Select Expression and enter node_ntp_offset_seconds
  5. Click Create
  6. Configure a Notifier for the A set of alerts for node Alert Group, by clicking the elipses for this Alert Group, and configuring the desired notifier in the Alert section at the bottom of the form.

Further Reading

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000020118
  • Creation Date: 06-May-2021
  • Modified Date:06-May-2021
    • SUSE Rancher

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center