Resolving "Conflict. The container name is already in use" errors when updating a Rancher Kubernetes Engine (RKE) CLI or Rancher v2.x provisioned Kubernetes cluster

Issue

When invoking rke up, to update a Rancher Kubernetes Engine (RKE) CLI provisioned Kubernetes cluster, or using Rancher to update a Rancher v2.x provisioned cluster, you encounter a Conflict. The container name is already in use error, of the following format:

Format of error from RKE CLI, for RKE provisioned clusters

FATA[0219] [file-deploy] Failed to deploy file [/etc/kubernetes/audit-policy.yaml] on node [172.27.6.22]: Failed to create [file-deployer] container on host [172.27.6.22]: Failed to create Docker container [file-deployer] on host [172.27.6.22]: Error response from daemon: Conflict. The container name "/file-deployer" is already in use by container "66b777d981aa0b0a9d6bc73e381e0f2bc8fc33ec00926aa0db51347607f8fcf8". You have to remove (or rename) that container to be able to reuse that name.

Format of error in Rancher UI, for Rancher v2.x provisioned clusters

[Failed to create Certificates deployer container on host [172.27.3.21]: Failed to create Docker container [cert-deployer] on host [172.27.3.21]: Error response from daemon: Conflict. The container name "/cert-deployer" is already in use by container "c3b35c454d6000266098573949d021f45b13a3c7f7306d7fdb58a5766f2f3312". You have to remove (or rename) that container to be able to reuse that name.]

Pre-requisites

• A Rancher Kubernetes Engine (RKE) CLI, or Rancher v2.x provisioned Kubernetes cluster
• SSH access to affected node(s) and Docker CLI access

Root cause

This error can occur when an rke up invocation or Rancher cluster update process is interrupted, leaving temporary cluster deployment containers on nodes.

Resolution

1. SSH into the node.
2. Remove stuck containers:

   docker rm -f file-deployer cert-deployer

3. Trigger a cluster update:
   • For RKE CLI provisioned clusters, invoke rke up.
   • For Rancher provisioned clusters:
     • Browse to the cluster in the Rancher UI
     • Click Edit from the action menu
     • Click Edit as YAML
     • Find the option addon_job_timeout and edit the value, incrementing it by one
     • Click Save