Rancher Monitoring: Known Issues with NFS Persistent Storage for Prometheus

This document (000021332) is provided subject to the disclaimer at the end of this document.

Environment

Rancher v2.x, from v2.5 and above
rancher-monitoring deployed with a Prometheus persistent storage configuration using NFS

Situation

The Prometheus server is unstable, with storage-related errors in the Prometheus Pod logs:

2022-05-26T07:15:19.251601279Z level=error ts=2022-05-26T07:15:19.251Z caller=db.go:745 component=tsdb msg="compaction failed" err="plan compaction: open /prometheus/01F8Q3MT8CJXMNMPKW7G86M6M5/meta.json: no such file or directory"

and/or a warning about NFS storage not being supported:

2022-05-25T10:15:40.601740700Z level=warn ts=2022-05-25T10:15:40.601Z caller=main.go:756 fs_type=NFS_SUPER_MAGIC msg="This filesystem is not supported and may lead to data corruption and data loss. Please carefully read https://prometheus.io/docs/prometheus/latest/storage/ to learn more about supported filesystems."

Resolution

As described in the Prometheus docs, NFS is not a supported filesystem for Prometheus database storage:

CAUTION: Non-POSIX compliant filesystems are not supported for Prometheus' local storage as unrecoverable corruptions may happen. NFS filesystems (including AWS's EFS) are not supported. NFS could be POSIX-compliant, but most implementations are not. It is strongly recommended to use a local filesystem for reliability.

You should therefore remove the persistent storage configuration using NFS in the rancher-monitoring chart options or switch to a supported persistent storage type, such as a block storage volume provisioned (e.g. AWS EBS, Longhorn, vSphere volumes).

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.