'Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io"' error when upgrading Rancher immediately after performing a local cluster Kubernetes upgrade

This document (000021315) is provided subject to the disclaimer at the end of this document.

Environment

Rancher v2.6.x
An RKE1 or RKE2 local cluster

Situation

The Rancher upgrade task is launched immediately after upgrading the underlying local cluster. The Rancher upgrade fails with the following error:
image.png

Resolution

Before running a Rancher upgrade, check that ingress-nginx Pods in the local cluster are running. If the Rancher upgrade task is executed in an automated fashion, a delay should be introduced between the cluster upgrade and the Rancher helm upgrade.

The commands below will enable you to validate that the ingress-nginx pods are running before performing the Rancher helm upgrade. The flag "timeout" ensures the command exits with an error on a timeout, rather than waiting indefinitely, e.g. --timeout=2m.
  • RKE1
kubectl rollout status daemonset -n ingress-nginx nginx-ingress-controller --timeout=x

kubectl wait --for=condition=Available -n ingress-nginx nginx-ingress-controller
  • RKE2
kubectl rollout status daemonset -n kube-system rke2-ingress-nginx-controller  --timeout=x

kubectl wait --for=condition=Available -n kube-system rke2-ingress-nginx-controller 

 

Cause

The 'helm upgrade --install` task for Rancher was invoked before the ingress-nginx pods in the cluster were in a running state, following the cluster upgrade. As a result, the ingress-nginx validating webhook could not be satisfied, resulting in these errors when helm attempted to patch the Rancher ingress resource.

Additional Information

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000021315
  • Creation Date: 29-Dec-2023
  • Modified Date:05-Feb-2024
    • SUSE Rancher

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center