Prometheus Node Exporter doesn't start after upgrade to SUSE Enterprise Storage 7

This document (000019871) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 7

Situation

After an upgrade from to SUSE Enterprise Storage 7, Prometheus' node-exporter daemons don't start. "ceph adm shell orch ls" prints the events:

service:node-exporter [ERROR] "Failed while placing node-exporter.NODE on NODE: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.NODE ...
Verifying port 9100 ...
ERROR: TCP Port(s) '9100' required for node-exporter already in use"

Resolution

Try to terminate any still running node-exporter processes and uninstall its RPM:
  1.   systemctl disable --now prometheus-node_exporter
  2.   zypper rm golang-github-prometheus-node_exporter
  3.   killall -9 node_exporter
Next, check if anything is anything is still listening on port 9100:
  ss -tapn '( sport = :9100 )'
and terminate it if that is the case.

Next, verify that IPv6 is enabled:
  grep -q TCPv6 /proc/net/protocols && echo "IPv6 enabled"
This should print "IPV6 enabled". If it doesn't, enable IPv6 support on your node(s).

Repeat these steps for all affected nodes.

Finally, restart the node-exporter daemons by running
  cephadm shell ceph orch --refresh

Cause

If there is some process already listening on port 9100 or if IPv6 has been disabled, prometheus-node-exporter is not able to bind(2) to the TCP port used to export the collected metrics to a Prometheus instance. The node exporter terminates with an error message in that case.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000019871
  • Creation Date: 10-Feb-2021
  • Modified Date:11-Feb-2021
    • SUSE Enterprise Storage

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center