Reduce Memory and CPU footprint of Prometheus Monitoring Operator

This document (000020733) is provided subject to the disclaimer at the end of this document.

Environment

rancher-monitoring: 100.1.3+up19.0.3

Situation

On a fresh install of rancher-monitoring, the Prometheus monitoring operator consumes high CPU and Memory resources without any custom Prometheus CRDs configured. You would notice the following error messages on the Prometheus Operator Pod very frequently.

$ kubectl logs -n cattle-monitoring-system rancher-monitoring-operator-784c69bc54-dvndg -f
level=info ts=2022-07-26T09:34:55.46606005Z caller=operator.go:1224 component=prometheusoperator msg="sync prometheus" key=cattle-monitoring-system/rancher-monitoring-prometheus
level=info ts=2022-07-26T09:34:55.612269913Z caller=operator.go:747 component=alertmanageroperator msg="sync alertmanager" key=cattle-monitoring-system/rancher-monitoring-alertmanager
level=info ts=2022-07-26T09:34:55.694485011Z caller=operator.go:747 component=alertmanageroperator msg="sync alertmanager" key=cattle-monitoring-system/rancher-monitoring-alertmanager
level=info ts=2022-07-26T09:34:55.92009322Z caller=operator.go:1224 component=prometheusoperator msg="sync prometheus" key=cattle-monitoring-system/rancher-monitoring-prometheus
level=info ts=2022-07-26T09:34:59.042606472Z caller=operator.go:747 component=alertmanageroperator msg="sync alertmanager" key=cattle-monitoring-system/rancher-monitoring-alertmanager
level=info ts=2022-07-26T09:34:59.043983987Z caller=operator.go:1224 component=prometheusoperator msg="sync prometheus" key=cattle-monitoring-system/rancher-monitoring-prometheus

Resolution

Add SecretListWatchSelector to reduce memory and CPU footprint.

prometheusOperator:
   secretFieldSelector: "type!=helm.sh/release.v1"

Status

Top Issue

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.