How to add custom labels to Alerts in Monitoring v2

A Rancher v2.5+ managed cluster, with the rancher-monitoring chart installed

Alerts in Monitoring v2 contain standard labels. In some cases, users may want to inject custom labels, such as Kubernetes cluster names to easily identify the environment or other custom labels for Alert routing.

Use the value defaultRules.additionalRuleLabels  in the rancher-monitoring chart to inject custom labels.

For example, to inject cluster name, open the Monitoring App in Apps from Cluster explorer. Under "Edit as YAML", add the custom label as below:

defaultRules:
  additionalRuleLabels:
    cluster: "My_Test_cluster"

Then click on "Deploy" or "Upgrade" if App is already installed.

If your receiver is webhook, then the alerts will have the custom labels as shown in the below example alert.

...
...
status":"firing",
"labels":{
  "alertname":"NodeClockNotSynchronising",
  "cluster":"My_Test_cluster", <<<------
  "container":"node-exporter",
  "endpoint":"metrics",
  "instance":"192.168.110.157:9796",
  "job":"node-exporter",
  "namespace":"cattle-monitoring-system",
  "pod":"rancher-monitoring-prometheus-node-exporter-lg2g6",
  "prometheus":"cattle-monitoring-system/rancher-monitoring-prometheus",
  "service":"rancher-monitoring-prometheus-node-exporter",
...
...