Incomplete Removal of Proxy Setting Variable by Rancher on Rancher provisioned downstream cluster.

Rancher version: 2.9.x or 2.10.x

Following the decommissioning of a proxy server and the subsequent removal of its configurations from the Rancher UI via the "Cluster Config > Agent Environment Variables" for Rancher provisioned downstream clusters, a failure to propagate these changes to the downstream nodes will result in image pull failures.

These failures may manifest as ImagePullBackOff errors or specific authentication errors such as "Error response from daemon: Get "https://registry.xyz.com/v2/": Forbidden" since the request to the Private Registry happens via the Proxy server which does not exist anymore. 

Upon removing Proxy settings from the Rancher UI for downstream clusters via "Cluster Config > Agent Environment Variables", verification of proxy setting removal on each downstream node is required if the image pull still continues to use Proxy configurations to establish connection to Private Registry. Inspect the /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env file for any remaining proxy configurations. If present, this signifies a failure of the rancher-system-agent to update its configuration from the upstream Rancher instance.

In this scenario, examine /etc/systemd/system/rancher-system-agent.env and /etc/systemd/system/rancher-system-agent.service for proxy settings. If found, these settings must be removed, and the rancher-system-agent service, along with the rke2-server or rke2-agent service, should be restarted to enforce the updated configuration.