Cluster and Project monitoring fail to deploy with restricted PodSecurityPolicy in Rancher v2.2

This document (000020227) is provided subject to the disclaimer at the end of this document.

Situation

Issue

Attempting to enable Cluster or Project monitoring, in a Rancher v2.2 cluster, where the restricted PodSecurityPolicy (PSP) is configured on the Cluster or Project, fails with the Grafana and Prometheus Pods in a CrashLoopBackOff.

The grafana-proxy container in the grafana-cluster-monitoring/grafana-project-monitoring Deployment and the promtheus-proxy container in the prometheus-cluster-monitoring/prometheus-project-monitoring StatefulSet fail with an error of the following format:

2019/09/20 11:54:17 [warn] 1#1: duplicate MIME type "text/html" in /var/run/nginx.conf:46
nginx: [warn] duplicate MIME type "text/html" in /var/run/nginx.conf:46
2019/09/20 11:54:17 [emerg] 1#1: chown("/tmp/nginx", 100) failed (1: Operation not permitted)
nginx: [emerg] chown("/tmp/nginx", 100) failed (1: Operation not permitted)

The grafana container in the grafana-cluster-monitoring/grafana-project-monitoring Deployment fails to start with an error of the following format shown in the events for the Pod:

Error: failed to start container "grafana": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/kubelet/pods/6c9bbb63-db9a-11e9-932e-2af11a72a258/volume-subpaths/grafana-static-contents/grafana/1\\\" to rootfs \\\"/var/lib/docker/overlay2/bec56dbc35983bd46debc3b8f1e7d88227556db353356695647d44a09a686eb2/merged\\\" at \\\"/var/lib/docker/overlay2/bec56dbc35983bd46debc3b8f1e7d88227556db353356695647d44a09a686eb2/merged/usr/share/grafana/public/app/plugins/datasource/prometheus/plugin.json\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

Pre-requisites

  • A cluster managed by Rancher v2.2
  • Cluster or Project monitoring enabled
  • PodSecurityPolicy enabled in the cluster and the restricted PSP configured at the cluster level, or on the Project for which monitoring is enabled

Root Cause

The monitoring system-charts in Rancher v2.2 are not compatible with the restricted PodSecurityPolicy. As a result where the restricted PSP is configured, monitoring will fail to deploy successfully.

Workaround

To workaround the impact to cluster monitoring, the following Role and RoleBinding can be applied to the cluster, by copying these to a file and applying with kubectl --config <cluster kubeconfig> apply -f <file>.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: default-psp-role
  namespace: cattle-prometheus
rules:
- apiGroups:
  - extensions
  resourceNames:
  - default-psp
  resources:
  - podsecuritypolicies
  verbs:
  - use

---

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: default-psp-rolebinding
  namespace: cattle-prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: default-psp-role
subjects:
- kind: Group
  name: system:serviceaccounts:cattle-prometheus
  apiGroup: rbac.authorization.k8s.io

Resolution

An update to ensure the monitoring system-charts are compatible with the restricted PodSecurityPolicy is available in Rancher v2.3.0 and above.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000020227
  • Creation Date: 06-May-2021
  • Modified Date:06-May-2021
    • SUSE Rancher

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center