Incomplete Removal of Proxy Setting Variable by Rancher on Rancher provisioned downstream cluster.
This document (000021825) is provided subject to the disclaimer at the end of this document.
Environment
Rancher version: 2.9.x or 2.10.x
Situation
Following the decommissioning of a proxy server and the subsequent removal of its configurations from the Rancher UI via the "Cluster Config > Agent Environment Variables" for Rancher provisioned downstream clusters, a failure to propagate these changes to the downstream nodes will result in image pull failures.
These failures may manifest as ImagePullBackOff
errors or specific authentication errors such as "Error response from daemon: Get "https://registry.xyz.com/v2/": Forbidden" since the request to the Private Registry happens via the Proxy server which does not exist anymore.
Resolution
Upon removing Proxy settings from the Rancher UI for downstream clusters via "Cluster Config > Agent Environment Variables", verification of proxy setting removal on each downstream node is required if the image pull still continues to use Proxy configurations to establish connection to Private Registry. Inspect the /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env
file for any remaining proxy configurations. If present, this signifies a failure of the rancher-system-agent to update its configuration from the upstream Rancher instance.
In this scenario, examine /etc/systemd/system/rancher-system-agent.env
and /etc/systemd/system/rancher-system-agent.service
for proxy settings. If found, these settings must be removed, and the rancher-system-agent
service, along with the rke2-server
or rke2-agent
service, should be restarted to enforce the updated configuration.
Disclaimer
This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.
- Document ID:000021825
- Creation Date: 06-May-2025
- Modified Date:22-May-2025
-
- SUSE Rancher
For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com