rancher-backup vs. etcd snapshot

SUSE Rancher +2.5.x

1. Purpose of each backup method:

Rancher Backup Operator [1]: This is an application-level backup tool specifically designed to backup and restore the Rancher application itself. It captures Rancher-specific resources and Custom Resource Definitions (CRDs) as YAML files, which can then be imported into another Rancher instance. This tool focuses on preserving the Rancher state and its related resources. 

etcd Snapshots [2]: These back up the entire etcd database, which serves as the "brain" of a Kubernetes cluster and the backing store for all cluster data. Etcd snapshots preserve the complete state of the Kubernetes cluster, including all configurations, workloads, and resources. 

2. When to use Rancher Backup:

The Rancher Backup Operator is ideal in these scenarios: 

• When you need to migrate your Rancher instance to another Kubernetes cluster
• For backing up just the Rancher-specific resources without the entire cluster data
• When planning a Rancher upgrade and want a safety net for your Rancher configuration
• For creating regular backups of your Rancher deployment that can be restored on a fresh cluster

It's highly recommended to use Rancher Backup for your local/upstream cluster (where Rancher itself is installed) to enable restoration or migration of Rancher when needed. 

3. When to use etcd Snapshots:

Etcd snapshots are the preferred option when:

• Recovering from a disaster that affects the entire Kubernetes cluster
• Rolling back Kubernetes to a previous version after a failed upgrade
• Backing up downstream clusters managed by Rancher
• Needing to restore the entire cluster state, including all non-Rancher resources
• Performing routine backups of your complete Kubernetes environment

Etcd snapshots are recommended for downstream clusters since Rancher Backup doesn't contain the Kubernetes cluster data stored on these clusters' databases.

4. Can/should both be used? 

Yes, both methods can be combined as they serve different purposes:

• Use Rancher Backup for your local/upstream cluster to protect the Rancher application itself
• Use etcd snapshots for your downstream clusters to protect their Kubernetes data

For the local/upstream cluster where Rancher is installed, you can also use etcd snapshots, but consider the following:

• If the issue is with your Rancher deployment specifically, use Rancher Backup
• If the issue is with the underlying Kubernetes cluster, it may be more efficient to build a new cluster and use Rancher Backup to migrate your Rancher deployment to it

A comprehensive backup strategy would include both types of backups for complete protection.

5. Comparison Table: 

6. Tips for storing backups to S3, restore implications, etc:

S3 Storage Recommendations:

• Configure an S3 location to save both the snapshots and the Rancher operator backups
• This approach ensures backups remain accessible even if something catastrophic happens to the cluster
• S3 provides durability and availability for your critical backup data

Backup Implementation Options:

• RKE2 etcd snapshots
• k3s etcd snapshots

Best Practices:

• Regularly test your backup and restore procedures to ensure they work as expected
• Document your backup strategy and restoration processes
• Establish a regular backup schedule for both Rancher Backup and etcd snapshots
• Store backups off-site or in a different region to protect against regional disasters

By implementing both backup methods according to their strengths, you create a comprehensive backup strategy that provides protection for both your Rancher application and your Kubernetes clusters.

References;

[1] https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery#installing-the-rancher-backup-operator
[2] https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/