How to enable antiAffinity for Rancher v2.x server pods

Task

By default the Rancher server pods are deployed without podAntiAffinity rules. As a result of this multiple Rancher pods may be scheduled onto a single node, potentially leading to temporary service disruption if the node is unavailable or gets rebooted.

Pre-requisites

• Running Rancher v2.x
• kubectl access to the cluster
• Rancher Kubernetes Engine (RKE) to be installed, with access to the cluster.yml and correspoding cluster.rkestate file see the RKE documentation for more information
• helm v3

Steps

You need to add the option --set-string antiAffinity=required to your Rancher install. Details on how to add this option to both new installations of Rancher, as well as existing deployment are provided below.

New Rancher installation

For new installations of Rancher, add the antiAffinity option to the helm install command, per the following example:

helm install rancher rancher-latest/rancher \
--namespace cattle-system \
--set hostname=mmattox-example.support.rancher.space \
--version 2.3.6 \
--set-string antiAffinity=required

NOTE: The Rancher version is pinned with the --version flag to prevent a version upgrade.

Update existing Rancher deployments

To add the antiAffinity option to an existing deployment of Rancher, follow the Rancher upgrade documentation, using the --version flag to pin to the running Rancher version, preventing a version upgrade.

1. Run helm get values rancher to get the current Rancher helm chart values, which will be used to generate the helm upgrade command with matching values.
2. Generate and run the helm upgrade command with the chart values, including the pinned version and antiAffinity option, per the following example:

   helm upgrade rancher rancher-stable/rancher \
   --namespace cattle-system \
   --set hostname=mmattox-example.support.rancher.space \
   --version 2.3.6 \
   --set-string antiAffinity=required

NOTE: The Rancher version is pinned with the --version flag to prevent a version upgrade. NOTE: We recommend saving this command for future Rancher upgrades to save time.

Verification

Run the command kubectl get deployment -n cattle-system rancher -o yaml and verify the following podAntiAffinity spec has been added:

[...]
spec:
 affinity:
   podAntiAffinity:
     requiredDuringSchedulingIgnoredDuringExecution:
     - labelSelector:
         matchExpressions:
         - key: app
           operator: In
           values:
           - rancher
       topologyKey: kubernetes.io/hostname
[...]

Rollback

To remove the antiAffnitiy configuration you should remove the --set-string antiAffinity=required option from the helm upgrade command and re-run this, per the following example:

helm upgrade rancher rancher-stable/rancher \
--namespace cattle-system \
--set hostname=mmattox-example.support.rancher.space \
--version 2.3.6

NOTE: The Rancher version is pinned with the --version flag to prevent a version upgrade.