How to clean up orphaned objects for an already deleted downstream cluster

This document (000021229) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Rancher 2.7.x

Situation

If you see the following error messages in the Rancher pod logs after a downstream cluster has already been deleted, then you have orphaned objects in the local cluster that need to be deleted.

2022/09/02 21:03:21 [ERROR] error syncing 'p-rs52c/creator-project-owner': handler mgmt-auth-prtb-controller: clusters.management.cattle.io "c-m-pdpztlk9" not found, requeuing
2022/09/02 21:03:21 [ERROR] error syncing 'p-9htch/creator-project-owner': handler mgmt-auth-prtb-controller: clusters.management.cattle.io "c-m-tgzdjmqc" not found, requeuing

2022/09/17 12:23:12 [ERROR] error syncing 'c-m-rl4xbzsk/default-token': handler cluster-registration-token: clusters.management.cattle.io "c-m-rl4xbzsk" not found, requeuing
2022/09/17 12:23:12 [ERROR] error syncing 'c-m-p8s2rmjk/default-token': handler cluster-registration-token: clusters.management.cattle.io "c-m-p8s2rmjk" not found, requeuing

2022/09/17 12:55:55 [ERROR] error syncing 'c-m-tgzdjmqc/creator-cluster-owner': handler mgmt-auth-crtb-controller: clusters.management.cattle.io "c-m-tgzdjmqc" not found, requeuing
2022/09/17 12:55:55 [ERROR] error syncing 'c-m-v5bkkkmm/creator-cluster-owner': handler mgmt-auth-crtb-controller: clusters.management.cattle.io "c-m-v5bkkkmm" not found, requeuing

Resolution

You will need to find the orphaned objects and then delete them. Below you will have the commands to find the objects first and then delete them

1. Orphaned projects:

2022/09/02 21:03:21 [ERROR] error syncing 'p-rs52c/creator-project-owner': handler mgmt-auth-prtb-controller: clusters.management.cattle.io "c-m-pdpztlk9" not found, requeuing
2022/09/02 21:03:21 [ERROR] error syncing 'p-9htch/creator-project-owner': handler mgmt-auth-prtb-controller: clusters.management.cattle.io "c-m-tgzdjmqc" not found, requeuing

kubectl get projects.management.cattle.io -A 
kubectl -n c-m-v5hx2h82 delete  projects.management.cattle.io p-7tsfg
kubectl -n c-m-v5hx2h82 delete  projects.management.cattle.io p-h57v9

2. Orphaned cluster registration token:

2022/09/17 12:23:12 [ERROR] error syncing 'c-m-rl4xbzsk/default-token': handler cluster-registration-token: clusters.management.cattle.io "c-m-rl4xbzsk" not found, requeuing
2022/09/17 12:23:12 [ERROR] error syncing 'c-m-p8s2rmjk/default-token': handler cluster-registration-token: clusters.management.cattle.io "c-m-p8s2rmjk" not found, requeuing

kubectl get clusterregistrationtokens.management.cattle.io -A 
kubectl -n c-m-v5hx2h82 delete clusterregistrationtokens.management.cattle.io default-token

3. Orphaned namespace:

2022/09/17 12:55:55 [ERROR] error syncing 'c-m-tgzdjmqc/creator-cluster-owner': handler mgmt-auth-crtb-controller: clusters.management.cattle.io "c-m-tgzdjmqc" not found, requeuing
2022/09/17 12:55:55 [ERROR] error syncing 'c-m-v5bkkkmm/creator-cluster-owner': handler mgmt-auth-crtb-controller: clusters.management.cattle.io "c-m-v5bkkkmm" not found, requeuing

kubectl delete namespace c-m-v5hx2h82

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.