Kernel live patching Kubernetes nodes using Rancher’s system-upgrade-controller

Rancher’s system-upgrade-controller allows to upgrade Kubernetes nodes using privileged pods within that Kubernetes cluster. This can be very helpful to install latest updates to fix CVE or to upgrade used software to a later version. But sometimes those updates require the system to reboot to take effect.

This might be no problem for most of the workloads running in pods on that machine. But what if your workload isn’t stateless and its rescheduling would lead to outages of your services?

SUSE’s Kernel live patching might help here as the Kernel modules of the Kubernetes node can be updated without requiring a reboot of the machine and without forcing the pods to be rescheduled.

How to?

Prerequisites

• A Kubernetes cluster with SUSE Linux Enterprise Server as the node OS
• SUSE Kernel live patching activated on those nodes *

Check if Kernel live patching is enabled

To verify if kernel live patching is enabled, run the following command on each of your Kubernetes nodes:

 $ klp status

This should return “ready”.

If it does not, check out https://documentation.suse.com/sles/15-SP1/html/SLES-all/cha-klp.html and follow the instructions and make sure Kernel live patching is enabled for your machines.

Deploy Rancher’s system-upgrade-controller

Donwload the latest version of the system-upgrade-controller from https://raw.githubusercontent.com/rancher/system-upgrade-controller/master/manifests/system-upgrade-controller.yaml .
To deploy, run:

$ kubectl apply -f system-upgrade-controller.yaml

To check if the deployment was successful, run:

$ kubectl get pods -n system-upgrade

If the deployment succeeded, the output may look like:

Deploy a Plan to upgrade the desired Kubernetes nodes

The system-upgrade-controller allows to apply Plans to the cluster to define what updates to take on the nodes and to select the nodes to be updated.

An example Plan looks like:

---
apiVersion: v1
kind: Secret
metadata:
  name: worker
  namespace: system-upgrade
type: Opaque
stringData:
  upgrade.sh: |
    #!/bin/sh
    set -e
    secrets=$(dirname $0)
    zypper patch -r SLE-Module-Live-Patching15-SP2-Updates -y

---
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: worker
  namespace: system-upgrade
spec:
  concurrency: 2
  nodeSelector:
    matchExpressions:
      - {key: node-role.kubernetes.io/worker, operator: Exists}
  serviceAccountName: system-upgrade
  secrets:
    - name: worker
      path: /host/run/system-upgrade/secrets/worker
  version: worker
  upgrade:
    image: registry.suse.com/suse/sle15:15.2
    command: ["chroot", "/host"]
    args: ["sh", "/run/system-upgrade/secrets/worker/upgrade.sh"]

The upper part (blue) deploys the Kubernetes Secret which just defines a script to be executed on the Node. In this example, the main line here is

zypper patch -r SLE-Module-Live-Patching15-SP2-Updates -y

which will apply all available kernel live patches for SLES 15 SP2.

The lower part (green) shows the Plan. The nodeSelector is used to define which Kubernetes nodes should apply the upgrade. It’s also possible to choose how many Nodes to be updated the same time (concurency). It is important that the image defined in the Plan matches the OS on the Kubernetes nodes. In the excerpt here, SLES 15 SP2 is used.

For more detailed information about the options, please visit https://github.com/rancher/system-upgrade-controller.

After deploying the Plan and the related Secret, the system-upgrade-controller will launch new pods on all nodes that match the nodeSelector. In the excerpt above, all nodes holding a label “node-role.kubernetes.io/worker” will be updated.

Enjoy!

You can now watch your workload related pods remain working while the nodes are getting updated.

* It is also possible to register SUSE Kernel live patching using Rancher’s system-upgrade controller. To do so, add the registration and installation  to the Secret. This then may look like:

---
apiVersion: v1
kind: Secret
metadata:
  name: worker
  namespace: system-upgrade
type: Opaque
stringData:
  upgrade.sh: |
    #!/bin/sh
    set -e
    secrets=$(dirname $0)
    SUSEConnect -r <live-patching-subscription-code>
    SUSEConnect -p sle-module-live-patching/15.2/x86_64 -r <live-patching-subscription-code>
    zypper install -y -t pattern lp_sles
    zypper patch -r SLE-Module-Live-Patching15-SP2-Updates -y