How to rotate rancher-webhook (cattle-webhook-tls) certificates?

This document (000020415) is provided subject to the disclaimer at the end of this document.

Environment

Rancher versions ≤ 2.6.2

Situation

An upgrade of Rancher or editing roles in Rancher UI fails with the below error.

Internal error occurred: failed calling webhook "rancherauth.cattle.io": Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation?timeout=10s": x509: certificate has expired or is not yet valid: current time 2021-10-25T07:43:50Z is after 2021-10-06T20:20:47Z

Resolution

Set the kubectl context for the Rancher management cluster (local cluster).
Take the backup of existing secret

kubectl get secret -n cattle-system cattle-webhook-tls -o yaml > cattle-webhook-tls.yaml

Delete the secret that contains expired certificate

kubectl delete secret -n cattle-system cattle-webhook-tls

Delete the rancher.cattle.io mutating webhook

kubectl delete mutatingwebhookconfigurations.admissionregistration.k8s.io --ignore-not-found=true rancher.cattle.io

Delete the rancher webhook Pod to regenerate the expired certificate.

kubectl delete pod -n cattle-system -l app=rancher-webhook

Cause

This issue is caused by the expired certificate of the rancher webhook.

Additional Information

In Rancher v2.6.3 and up, rancher-webhook deployments will automatically renew their TLS certificate when it is within 30 or fewer days of its expiration date.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.