etcd snapshots showing 0B size in the Rancher UI

This document (000021447) is provided subject to the disclaimer at the end of this document.

Environment

Rancher managed RKE2 cluster

Situation

When viewing etcd snapshots in the Rancher UI for a downstream RKE2 cluster, the snapshots show a size of 0 B.

Resolution

The workaround is to modify the rke2 lease and make the lease-holder match the rke2-etcd lease. The following one-liner can be used for implementing this workaround:

kubectl get lease -n kube-system rke2-etcd -o jsonpath='{.spec.holderIdentity}' | xargs -I {} kubectl patch lease -n kube-system  --patch='{"spec":{"holderIdentity":"{}"}}' --type=merge rke2; kubectl get lease -n kube-system -o custom-columns='name:.metadata.name,holder:.spec.holderIdentity' | grep rke2

After the workaround is implemented, please wait some time for the snapshots to reconcile.

Ultimately, the solution will be to upgrade to the version where this bug is fixed.

Cause

One reason this can happen is when there is a mismatch between the lease-holder for the rke2 and rke2-etcd leases:

$ kubectl get lease -n kube-system
NAME                                        HOLDER                                                                           AGE
cattle-controllers                          cattle-cluster-agent-57cc5dc68f-ddspd                                            26h
kube-apiserver-6gxa3fbniearwl6wdxdj6usnae   kube-apiserver-6gxa3fbniearwl6wdxdj6usnae_de66a659-04a1-40db-9061-63a1c4f3d2bc   54m
kube-apiserver-namyzjfjup76dymtfup4u2yxpy   kube-apiserver-namyzjfjup76dymtfup4u2yxpy_b08419af-5859-4270-9f3a-fef4bfeff20f   26h
kube-controller-manager                     ip-10-0-0-213_3af6132f-80f2-47ca-baa9-a4f5c4bcebc6                               26h
kube-scheduler                              ip-10-0-0-213_ee9eeff7-f8a7-46b2-80f3-e4c7ccae9631                               26h
rke2                                        ip-10-0-0-136                                                                    26h
rke2-cloud-controller-manager               ip-10-0-0-213_05e8dded-712d-4606-9c85-dfb60c7c9c59                               26h
rke2-etcd                                   ip-10-0-0-213                                                                    26h
rke2-ingress-nginx-leader                   rke2-ingress-nginx-controller-bwbmh                                              26h
snapshot-controller-leader                  rke2-snapshot-controller-6f7bbb497d-mqrxt                                        26h

This is a known bug: https://github.com/rancher/rke2/issues/5866

Additional Information

If there is no lease-holder mismatch, some other potential issues that could cause this problem are mentioned in the following knowledge base articles:

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.