'Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io"' error when upgrading Rancher immediately after performing a local cluster Kubernetes upgrade

This document (000021315) is provided subject to the disclaimer at the end of this document.

Environment

Rancher v2.6.x
An RKE1 or RKE2 local cluster

Situation

The Rancher upgrade task is launched immediately after upgrading the underlying local cluster. The Rancher upgrade fails with the following error:

Resolution

Before running a Rancher upgrade, check that ingress-nginx Pods in the local cluster are running. If the Rancher upgrade task is executed in an automated fashion, a delay should be introduced between the cluster upgrade and the Rancher helm upgrade.

The commands below will enable you to validate that the ingress-nginx pods are running before performing the Rancher helm upgrade. The flag "timeout" ensures the command exits with an error on a timeout, rather than waiting indefinitely, e.g. --timeout=2m.

RKE1

kubectl rollout status daemonset -n ingress-nginx nginx-ingress-controller --timeout=x

kubectl wait --for=condition=Available -n ingress-nginx nginx-ingress-controller

RKE2

kubectl rollout status daemonset -n kube-system rke2-ingress-nginx-controller  --timeout=x

kubectl wait --for=condition=Available -n kube-system rke2-ingress-nginx-controller

Cause

The 'helm upgrade --install` task for Rancher was invoked before the ingress-nginx pods in the cluster were in a running state, following the cluster upgrade. As a result, the ingress-nginx validating webhook could not be satisfied, resulting in these errors when helm attempted to patch the Rancher ingress resource.

Additional Information

Rancher upgrade checklist
- https://www.suse.com/es-es/support/kb/doc/?id=000020061
Rancher upgrade
- https://ranchermanager.docs.rancher.com/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/upgrades#steps-to-upgrade-rancher
How to install or upgrade to a specific Rancher v2.x version
- https://www.suse.com/es-es/support/kb/doc/?id=000020217

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.