Removing the global system-default-registry value

This document (000020658) is provided subject to the disclaimer at the end of this document.

Situation

When removing the global system-default-registry setting in order to use a cluster-level private registry with authentication, the order in which you make these changes is important. If you remove the global system-default-registry setting first, this could cause downstream clusters to go into an error state showing the following error in the Rancher UI:

Cluster must have at least one etcd plane host: failed to connect to the following etcd host(s) [IP Address]

Resolution

Set the private registry at the cluster-level first and wait for the downstream cluster to finish updating before removing the value from the global system-default registry.

If the value for the global system-default-registry was removed first and the downstream cluster is in an error state, the following should resolve the issue:

From the Rancher (local) management cluster edit each of the control plane/etcd nodes in the downstream cluster with the following command:

kubectl edit node.management.cattle.io -n <clusterID> <machineID>

Locate the nodePlan field, to find the image field and append the proper registry to the start of the value
Update the cluster-level private registry

Cause

If the global system-default-registry value is removed, the downstream cluster will begin updating and may not be able to pull the rancher/rancher-agent image causing the downstream cluster to not be able to communicate with the local Rancher cluster. Therefore, the downstream cluster should have the proper private registry set at the cluster-level before the global system-default-registry value is removed.

Additional Information

Global registry: https://rancher.com/docs/rancher/v2.6/en/admin-settings/config-private-registry/
Cluster-level registry: https://rancher.com/docs/rancher/v2.6/en/cluster-admin/editing-clusters/rke-config-reference/#private-registries

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.