How to recover after deleting the Calico CRDs from a cluster

This document (000020213) is provided subject to the disclaimer at the end of this document.

Situation

Issue

Calico uses a number of Custom Resource Definitions (CRDs) in order to store configuration data in Custom Resources. In the event that these CRDs are accidentally deleted from a cluster by a user, the configuration data in these Custom Resources will be deleted, preventing successful programming of pod networking. This article documents how to recreate the CRDs and ensure the configuration data is also re-populated.

Pre-requisites

A RKE1 supported cluster provisioned by the RKE CLI or Rancher v2.x, running with the Canal or Calico network providers

Resolution

1. Re-create the CRDs

The first step is to re-create the Custom Resource Definitions (CRDs). These definitions depend on the Kubernetes version running in the cluster and on whether the cluster uses the Canal or Calico network provider. If the cluster was provisioned using Rancher, follow these steps:

Delete the job rke-network-plugin-deploy-job in the kube-system namespace.
Edit the cluster configuration on Rancher Manager as follows:
- Navigate to the affected downstream cluster under Cluster Management and edit the Cluster Configuration.
- Go to Cluster Configuration > Advanced Options.
- Under Recurring etcd Snapshot Retention, increase the quantity by 1.
The above operation will initiate a cluster reconciliation, prompting a new rke-network-plugin-deploy-job to run and deploy the missing network plugin CRDs.

2. Delete a network pod to trigger re-creation of the Calico custom resources

Delete a network provider pod from a single node in the cluster, per the network provider specific instructions below. This will trigger creation of a new pod on that node, and the initialization of this will create the Calico custom resources containing Calico configuration. After this cluster networking should be fully restored.

Canal Network Provider

Delete one of the canal pods within the kube-system namespace.

Calico Network Provider

Delete one of the calico-node pods within the kube-system namespace.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.