Failed to handling tunnel request from remote address x.x.x.x:42412 (X-Forwarded-For: x.x.x.x): response 400: cluster not found

This document (000020950) is provided subject to the disclaimer at the end of this document.

Environment

  • Rancher v2.x

Situation

There are a lot of errors observed in the Rancher pods:
[ERROR] Failed to handling tunnel request from remote address x.x.x.x:42412 (X-Forwarded-For: x.x.x.x): response 400: cluster not found

Resolution

The error message "Failed to handling tunnel request from remote address x.x.x.x:42412 (X-Forwarded-For: x.x.x.x): response 400: cluster not found"  indicates that on some hosts from now-deleted clusters, there are Rancher agent containers/Pods that are still running and attempting to connect to Rancher.

To locate these instances and stop the running containers or hosts to prevent these messages:

Suppose your load-balancer is performing Layer-7 load-balancing and setting the X-Forwarded-For header itself. In that case, you could enable use-forwarded-headers on the ingress-nginx controller in the Rancher local cluster (https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#use-forwarded-headers).

This would pass through the X-Forwarded-For header from the loadbalancer, enabling you to identify the hosts from which these requests originate within the Rancher Pod logs.




 

Cause

Remaining nodes from deleted clusters trying to connect to Rancher.

Additional Information

The reason for seeing  IPs in the range 10.42.x.x in the remote address and your load-balancer IPs in the X-Forwarded-For on the logs is that:
  1.     the remote address is the ingress-nginx Pod IP
  2.     the X-Forwarded-For address is the load-balancer forwarding the requests to the cluster.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000020950
  • Creation Date: 23-Jan-2023
  • Modified Date:30-Jan-2023
    • SUSE Rancher

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center