How to add custom labels to Alerts in Monitoring v2

This document (000020302) is provided subject to the disclaimer at the end of this document.

Environment

Rancher Version: v2.5.8
Monitoring Chart Version: 14.5.100

Situation

Alerts in Monitoring v2 contain standard labels. But in some cases, users want to inject custom labels like Kubernetes cluster names to easily identify the environment or need the labels for further Alert routing.

Resolution

Use defaultRules.additionalRuleLabels in the Monitoring Apps's YAML spec to inject custom labels.

To inject cluster name, open the Monitoring App in Apps& Marketplace from Cluster explorer.
Under "Edit as YAML", add the custom label as below.

defaultRules:
  additionalRuleLabels:
    cluster: "My_Test_cluster"

Then click on "Deploy" or "Upgrade" if App is already installed.

If your receiver is webhook, then the alerts will have the custom labels as shown in the below example alert.

...
...
status":"firing",
"labels":{
  "alertname":"NodeClockNotSynchronising",
  "cluster":"My_Test_cluster", <<<------
  "container":"node-exporter",
  "endpoint":"metrics",
  "instance":"192.168.110.157:9796",
  "job":"node-exporter",
  "namespace":"cattle-monitoring-system",
  "pod":"rancher-monitoring-prometheus-node-exporter-lg2g6",
  "prometheus":"cattle-monitoring-system/rancher-monitoring-prometheus",
  "service":"rancher-monitoring-prometheus-node-exporter",
...
...

Additional Information

GitHub issue #3325 is opened to add additional labels via UI.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.