Troubleshooting downstream cluster disconnections and kubectl timeouts in Rancher 2.6.4 and 2.6.5

This document (000020745) is provided subject to the disclaimer at the end of this document.

Environment

Rancher Version 2.6.4 and 2.6.5 with multiple downstream clusters

Situation

Clusters will be unavailable or running kubectl commands from the built-in shell from the UI will timeout.

Examples:

kubectl get nodes
Unable to connect to the server: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
There are no log errors in rancher or downstream agents
Log nginx: upstream_status="-" upstream_response_time="32.000" request_uri="/k8s/clusters/[cluster]/apis?timeout=32s"

Resolution

To restart the Rancher Pods:

kubectl rollout restart deploy rancher -n cattle-system ; kubectl rollout status deploy rancher -n cattle-system

Cause

https://github.com/rancher/rancher/issues/37250

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.