How to regenerate Service Account tokens in Kubernetes

This document (000020019) is provided subject to the disclaimer at the end of this document.

Situation

Task

At times Service Account tokens could be rendered invalid. This could be due to restoring etcd backups or regenerating CA certificates within your cluster.

This problem manifests in clients unable to authenticate against the Kubernetes API, for both pods within the cluster, or system components like the controller-manager or kube-proxy. A telling sign would be errors in these clients of the format connect: connection refused or within the Kubernetes API Server logs, showing a large number of clients failing TLS authentication.

Steps

With an admin kubeconfig sourced for the cluster facing issues, run the command below, to generate the list of kubectl commands required to delete all Service Account token secrets. After running the provided kubectl commands from the output, you will need to recreate pods, e.g. by deleting them, in order to regenerate the Service Account token.

kubectl get secret --all-namespaces | awk '{ if ($3 == "kubernetes.io/service-account-token") print "kubectl -n", $1 " delete secret", $2 }'

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.