Rancher Upgrade Checklist

This document (000020061) is provided subject to the disclaimer at the end of this document.

Situation

Task

This article details the high-level steps required when planning and performing a Rancher and/or Kubernetes upgrade.

Planning

The following are the high-level rules for planning a Rancher/Kubernetes/Docker upgrade.

  • The support matrix gives a great idea of the versions that are certified by Rancher and work best together.
  • The recommended order of upgrades is: Rancher, Kubernetes, and then Docker/OS
  • All upgrades should be performed in a lab or non-prod environment first
  • Please see the following recommendations when planning version upgrades:
    • Rancher: perform one minor version jump at a time
      • For example: when upgrading from v2.1.x -> v2.3.x we encourage upgrading v2.1.x -> v2.2.x -> v2.3.x
    • Kubernetes: perform no more than 2 minor versions at a time, ideally avoid skipping minor versions entirely as this can increase the chances of an issue due to accumulated changes
      • For example: when upgrading from v1.13.x -> v1.16.x we encourage upgrading v1.13.x -> v1.14.x -> v1.15.x -> v1.16.x
    • RKE: perform one major RKE version jump at a time
      • For example: when upgrading from v0.1.x -> v1.1.0 instead do v0.1.x -> v0.2.x -> v.0.3.x -> v1.0.x -> v1.1.x
  • Adding a grace period between upgrades is recommended
    • For example: add 24 hours between upgrading Rancher, the local cluster, and downstream clusters
  • It is not required, but it is recommended to pause any application deployments using the Rancher API during an upgrade

Data collection

Before you start your upgrade, please collect the following pieces of information to best prepare yourself in case you need to open a support ticket.

  • Scheduled change window:
  • Current Rancher version (rancher/rancher image tag, or shown bottom left in the UI):
  • Target Rancher version:
  • Installation option (single install/HA):
  • Current Kubernetes version of Rancher local cluster (use kubectl version):
  • Current Docker version (use docker version):

Rancher Pre-Upgrade

  • Check if the Rancher UI is accessible
  • Check if all clusters in UI are in an Active state
  • Check if all pods in kube-system and cattle-system namespaces are running (in both Rancher and downstream clusters)
    kubectl get pods -n kube-system
    kubectl get pods -n cattle-system
  • Verify the datastore has scheduled snapshots configured, and these are working.
    • RKE: if Rancher is deployed on a Kubernetes cluster built with RKE, verify etcd snapshots are enabled and working, on etcd nodes you can confirm with the following:
      ls -l /opt/rke/etcd-snapshots
      docker logs etcd-rolling-snapshots
    • k3s: if Rancher is deployed on a k3s Kubernetes cluster, ensure scheduled backups are configured and working. Please see the k3s documentation pages for further information on this.
  • Create a one-time datastore snapshot, please see the following documentation for RKE and k3s, and the single node Docker install options for more information
  • RKE: check for expired/expiring Kubernetes certs
    for i in $(ls /etc/kubernetes/ssl/*.pem|grep -v key); do echo -n $i" "; openssl x509 -startdate -enddate -noout -in $i | grep 'notAfter='; done

Rancher Upgrade steps

Rancher Review/Verify

After the upgrade is completed, go through the following checklist to verify your environment is in working order.

  • Check if the Rancher UI is accessible
    • You should be able to login into Rancher, view clusters, and browse to workloads
  • Verify the Rancher version has changed in UI
    • After logging into Rancher, review the version in the bottom left corner of the page
  • Check if all clusters in UI are in an Active state
  • Check if all pods in kube-system and cattle-system are running (in both Rancher and downstream clusters)
  • Check the cattle-cluster-agent and cattle-node-agent pods are running in all downstream clusters and running the latest version
    • The rollout of the updated agent versions can take some time if there are a lot of downstream clusters or nodes
  • Create a one-time datastore snapshot, please see the following documentation for RKE and k3s, and the single node Docker install options for more information

Rancher Rollback steps

The Rancher rollback process is detailed in the rollback documentation, please follow the relevant link for Rancher installed on a Kubernetes cluster, or Docker

Follow-up tasks (optional)

  • Upgrade the Rancher management cluster, this is often a follow-up to the Rancher upgrade. Please see the RKE and k3s upgrade documentation for the upgrade process, as mentioned it is best to avoid skipping minor versions
  • Upgrade the downstream clusters, please see the documentation for more information. A snapshot of both the local and downstream clusters before the upgrade is recommended to provides the maximum amount of recoverability options in the event of a rollback
  • Docker/OS upgrades, please our article on performing rolling changes to nodes

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000020061
  • Creation Date: 20-Oct-2021
  • Modified Date:20-Oct-2021
    • SUSE Rancher

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center