Cluster and Project monitoring fail to deploy with restricted PodSecurityPolicy in Rancher v2.2
This document (000020227) is provided subject to the disclaimer at the end of this document.
Attempting to enable Cluster or Project monitoring, in a Rancher v2.2 cluster, where the restricted PodSecurityPolicy (PSP) is configured on the Cluster or Project, fails with the Grafana and Prometheus Pods in a CrashLoopBackOff.
grafana-proxy container in the
grafana-project-monitoring Deployment and the
promtheus-proxy container in the
prometheus-project-monitoring StatefulSet fail with an error of the following format:
2019/09/20 11:54:17 [warn] 1#1: duplicate MIME type "text/html" in /var/run/nginx.conf:46 nginx: [warn] duplicate MIME type "text/html" in /var/run/nginx.conf:46 2019/09/20 11:54:17 [emerg] 1#1: chown("/tmp/nginx", 100) failed (1: Operation not permitted) nginx: [emerg] chown("/tmp/nginx", 100) failed (1: Operation not permitted)
grafana container in the
grafana-project-monitoring Deployment fails to start with an error of the following format shown in the events for the Pod:
Error: failed to start container "grafana": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/kubelet/pods/6c9bbb63-db9a-11e9-932e-2af11a72a258/volume-subpaths/grafana-static-contents/grafana/1\\\" to rootfs \\\"/var/lib/docker/overlay2/bec56dbc35983bd46debc3b8f1e7d88227556db353356695647d44a09a686eb2/merged\\\" at \\\"/var/lib/docker/overlay2/bec56dbc35983bd46debc3b8f1e7d88227556db353356695647d44a09a686eb2/merged/usr/share/grafana/public/app/plugins/datasource/prometheus/plugin.json\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
- A cluster managed by Rancher v2.2
- Cluster or Project monitoring enabled
- PodSecurityPolicy enabled in the cluster and the restricted PSP configured at the cluster level, or on the Project for which monitoring is enabled
The monitoring system-charts in Rancher v2.2 are not compatible with the restricted PodSecurityPolicy. As a result where the restricted PSP is configured, monitoring will fail to deploy successfully.
To workaround the impact to cluster monitoring, the following Role and RoleBinding can be applied to the cluster, by copying these to a file and applying with
kubectl --config <cluster kubeconfig> apply -f <file>.
apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: default-psp-role namespace: cattle-prometheus rules: - apiGroups: - extensions resourceNames: - default-psp resources: - podsecuritypolicies verbs: - use --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: default-psp-rolebinding namespace: cattle-prometheus roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: default-psp-role subjects: - kind: Group name: system:serviceaccounts:cattle-prometheus apiGroup: rbac.authorization.k8s.io
An update to ensure the monitoring system-charts are compatible with the restricted PodSecurityPolicy is available in Rancher v2.3.0 and above.
This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.
- Document ID:000020227
- Creation Date: 06-May-2021
- Modified Date:06-May-2021
- SUSE Rancher
For questions or concerns with the SUSE Knowledgebase please contact: firstname.lastname@example.org