Task

This article details the high-level steps required when planning and performing a Rancher and/or Kubernetes upgrade.

Planning

The following are the high-level rules for planning a Rancher/Kubernetes/Docker upgrade.

• The support matrix gives a great idea of the versions that are certified by Rancher and work best together.
• The recommended order of upgrades is: Rancher, Kubernetes, and then Docker/OS
• All upgrades should be performed in a lab or non-prod environment first
• Please see the following recommendations when planning version upgrades:
  • Rancher: Do no skip any minor version when upgrading.
    • For example: when upgrading from v2.1.x -> v2.3.x we encourage upgrading v2.1.x -> v2.2.x -> v2.3.x
  • Kubernetes: Avoid skipping minor versions, as this can increase the chances of an issue due to accumulated changes.
    • For example: when upgrading from v1.13.x -> v1.16.x we encourage upgrading v1.13.x -> v1.14.x -> v1.15.x -> v1.16.x
  • RKE: perform one major RKE version jump at a time
    • For example: when upgrading from v0.1.x -> v1.1.0 instead do v0.1.x -> v0.2.x -> v.0.3.x -> v1.0.x -> v1.1.x
• Adding a grace period between upgrades is recommended
  • For example: add 24 hours between upgrading Rancher, the local cluster, and downstream clusters
• It is not required, but it is recommended to pause any application deployments using the Rancher API during an upgrade

Data collection

Before you start your upgrade, please collect the following pieces of information to best prepare yourself in case you need to open a support ticket.

• Scheduled change window:
• Current Rancher version (rancher/rancher image tag, or shown bottom left in the UI):
• Target Rancher version:
• Installation option (single install/HA):
• Current Kubernetes version of Rancher local cluster (use kubectl version):
• Current Docker version (use docker version):

Rancher Pre-Upgrade

• Check if the Rancher UI is accessible
• Check if all clusters in UI are in an Active state
• Check if all pods in kube-system and cattle-system namespaces are running (in both Rancher and downstream clusters)

  kubectl get pods -n kube-system
  kubectl get pods -n cattle-system

• Verify the datastore has scheduled snapshots configured, and these are working.
  • RKE: if Rancher is deployed on a Kubernetes cluster built with RKE, verify etcd snapshots are enabled and working, on etcd nodes you can confirm with the following:

    ls -l /opt/rke/etcd-snapshots
    docker logs etcd-rolling-snapshots

  • k3s: if Rancher is deployed on a k3s Kubernetes cluster, ensure scheduled backups are configured and working. Please see the k3s documentation pages for further information on this.
• Create a one-time datastore snapshot, please see the following documentation for RKE and k3s, and the single node Docker install options for more information
• RKE: check for expired/expiring Kubernetes certs

  for i in $(ls /etc/kubernetes/ssl/*.pem|grep -v key); do echo -n $i" "; openssl x509 -startdate -enddate -noout -in $i | grep 'notAfter='; done

Rancher Upgrade steps

• The Rancher upgrade process is detailed in the upgrade documentation for both HA, and single node using Docker.

Rancher Review/Verify

After the upgrade is completed, go through the following checklist to verify your environment is in working order.

• Check if the Rancher UI is accessible
  • You should be able to login into Rancher, view clusters, and browse to workloads
• Verify the Rancher version has changed in UI
  • After logging into Rancher, review the version in the bottom left corner of the page
• Check if all clusters in UI are in an Active state
• Check if all pods in kube-system and cattle-system are running (in both Rancher and downstream clusters)
• Check the cattle-cluster-agent and cattle-node-agent pods are running in all downstream clusters and running the latest version
  • The rollout of the updated agent versions can take some time if there are a lot of downstream clusters or nodes
• Create a one-time datastore snapshot, please see the following documentation for RKE and k3s, and the single node Docker install options for more information

Rancher Rollback steps

The Rancher rollback process is detailed in the rollback documentation, please follow the relevant link for Rancher installed on a Kubernetes cluster, or Docker

Follow-up tasks (optional)

• Upgrade the Rancher management cluster, this is often a follow-up to the Rancher upgrade. Please see the RKE and k3s upgrade documentation for the upgrade process, as mentioned it is best to avoid skipping minor versions
• Upgrade the downstream clusters, please see the documentation for more information. A snapshot of both the local and downstream clusters before the upgrade is recommended to provides the maximum amount of recoverability options in the event of a rollback
• Docker/OS upgrades, please our article on performing rolling changes to nodes