Resolving a fleet-agent that is stuck in the Pending-Upgrade state

This document (000020859) is provided subject to the disclaimer at the end of this document.

Situation

The fleet-agent is stuck in a "Pending-Upgrade" state and showing the following error:

time="2022-07-19T16:12:08Z" level=info msg="preparing upgrade for fleet-agent-c-97hcq" time="2022-07-19T16:12:08Z" level=info msg="getting history for release fleet-agent-c-97hcq" time="2022-07-19T16:12:08Z" level=error msg="error syncing 'cluster-fleet-default-c-97hcq-86145229ab95/fleet-agent-c-97hcq': handler bundle-deploy: another operation (install/upgrade/rollback) is in progress, requeuing"

Resolution

Run the following command against the cluster where the fleet-agent is running:

kubectl get secret -A -l status=pending-upgrade

It will show the output of a secret that is causing the pending-upgrade state as follows:

NAMESPACE             NAME                                        TYPE                 DATA   AGE
cattle-fleet-system   sh.helm.release.v1.fleet-agent-c-97hcq.v2   helm.sh/release.v1   1      132d

Based on the above output above, run through the following steps:
1. Backup the yaml (and save it to a persistent location) for the fleet secret that is causing the pending-upgrade state:

kubectl get secret -n cattle-fleet-system sh.helm.release.v1.fleet-agent-c-97hcq.v2 -oyaml > fleet-agent-c-97hcq.yaml

2. Delete the secret:

kubectl delete secret -n cattle-fleet-system sh.helm.release.v1.fleet-agent-c-97hcq.v2

3. In the Rancher UI, go to Continuous Delivery and do a "Force Update" on the downstream cluster in question

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.