Issue

Attempting to enable Cluster or Project monitoring, in a Rancher v2.2 cluster, where the restricted PodSecurityPolicy (PSP) is configured on the Cluster or Project, fails with the Grafana and Prometheus Pods in a CrashLoopBackOff.

The grafana-proxy container in the grafana-cluster-monitoring/grafana-project-monitoring Deployment and the promtheus-proxy container in the prometheus-cluster-monitoring/prometheus-project-monitoring StatefulSet fail with an error of the following format:

2019/09/20 11:54:17 [warn] 1#1: duplicate MIME type "text/html" in /var/run/nginx.conf:46
nginx: [warn] duplicate MIME type "text/html" in /var/run/nginx.conf:46
2019/09/20 11:54:17 [emerg] 1#1: chown("/tmp/nginx", 100) failed (1: Operation not permitted)
nginx: [emerg] chown("/tmp/nginx", 100) failed (1: Operation not permitted)

The grafana container in the grafana-cluster-monitoring/grafana-project-monitoring Deployment fails to start with an error of the following format shown in the events for the Pod:

Error: failed to start container "grafana": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/kubelet/pods/6c9bbb63-db9a-11e9-932e-2af11a72a258/volume-subpaths/grafana-static-contents/grafana/1\\\" to rootfs \\\"/var/lib/docker/overlay2/bec56dbc35983bd46debc3b8f1e7d88227556db353356695647d44a09a686eb2/merged\\\" at \\\"/var/lib/docker/overlay2/bec56dbc35983bd46debc3b8f1e7d88227556db353356695647d44a09a686eb2/merged/usr/share/grafana/public/app/plugins/datasource/prometheus/plugin.json\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

Pre-requisites

• A cluster managed by Rancher v2.2
• Cluster or Project monitoring enabled
• PodSecurityPolicy enabled in the cluster and the restricted PSP configured at the cluster level, or on the Project for which monitoring is enabled

Root Cause

The monitoring system-charts in Rancher v2.2 are not compatible with the restricted PodSecurityPolicy. As a result where the restricted PSP is configured, monitoring will fail to deploy successfully.

Workaround

To workaround the impact to cluster monitoring, the following Role and RoleBinding can be applied to the cluster, by copying these to a file and applying with kubectl --config <cluster kubeconfig> apply -f <file>.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: default-psp-role
  namespace: cattle-prometheus
rules:
- apiGroups:
  - extensions
  resourceNames:
  - default-psp
  resources:
  - podsecuritypolicies
  verbs:
  - use

---

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: default-psp-rolebinding
  namespace: cattle-prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: default-psp-role
subjects:
- kind: Group
  name: system:serviceaccounts:cattle-prometheus
  apiGroup: rbac.authorization.k8s.io

Resolution

An update to ensure the monitoring system-charts are compatible with the restricted PodSecurityPolicy is available in Rancher v2.3.0 and above.