Prometheus Node Exporter doesn't start after upgrade to SUSE Enterprise Storage 7

This document (000019871) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 7

Situation

After an upgrade from to SUSE Enterprise Storage 7, Prometheus' node-exporter daemons don't start. "ceph adm shell orch ls" prints the events:

service:node-exporter [ERROR] "Failed while placing node-exporter.NODE on NODE: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.NODE ...
Verifying port 9100 ...
ERROR: TCP Port(s) '9100' required for node-exporter already in use"

Resolution

Try to terminate any still running node-exporter processes and uninstall its RPM:

systemctl disable --now prometheus-node_exporter
zypper rm golang-github-prometheus-node_exporter
killall -9 node_exporter

Next, check if anything is anything is still listening on port 9100:
ss -tapn '( sport = :9100 )'
and terminate it if that is the case.

Next, verify that IPv6 is enabled:
grep -q TCPv6 /proc/net/protocols && echo "IPv6 enabled"
This should print "IPV6 enabled". If it doesn't, enable IPv6 support on your node(s).

Repeat these steps for all affected nodes.

Finally, restart the node-exporter daemons by running
cephadm shell ceph orch --refresh

Cause

If there is some process already listening on port 9100 or if IPv6 has been disabled, prometheus-node-exporter is not able to bind(2) to the TCP port used to export the collected metrics to a Prometheus instance. The node exporter terminates with an error message in that case.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.