Prometheus Exporter not able to scrape filesystem metrics from nodes when SELinux is enabled.

This document (000020747) is provided subject to the disclaimer at the end of this document.

Environment

Any Linux distribution with SELinux enabled would have this issue. 

Situation

Using Prometheus Graphs, when users are trying to query filesystem metrics from nodes, they would not see any metrics due to SELinux being enabled. SELinux policies reject the query due to insufficient privileges. 

PromQL Query: 

node_filesystem_*


Error on the Prometheus exporter pod:

level=error ts=2022-07-21T16:13:08.502Z caller=collector.go:169 msg="collector failed" name=filesystem duration_seconds=0.000837846 err="open /host/proc/1/mounts: permission denied" 

 

Resolution

This is not an issue with the Prometheus node exporter from the rancher monitoring chart but rather an issue from the SELinux side. There are three ways to verify that the SELinux team needs to be involved in fixing this issue. 

1.) Disable SELinux, and the Prometheus node exporter should be able to start querying the node-level metrics. (NOT RECOMMENDED)

2.) If SELinux cannot be disabled, you can modify the helm chart and update the Prometheus-node-exporter section with the below seLinuxOption called spc_t, which gives container super-privileged access. (NOT RECOMMENDED)
securityContext: 
    seLinuxOptions: 
      type: spc_t
3.) If super-privileged access cannot be provided to the container, ask the SELinux team to create necessary policies for processes to access the files that node-exporter pods are looking for. (RECOMMENDED APPROACH) 
 

Cause

We notice this issue with SELinux enabled cluster nodes.

Status

Top Issue

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000020747
  • Creation Date: 02-Sep-2022
  • Modified Date:06-Sep-2022
    • SUSE Rancher

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center