How to run workloads on etcd or controlplane nodes, without the worker role, in a Rancher Kubernetes Engine (RKE) or Rancher v2.x provisioned Kubernetes cluster

Task

Although it is usually not advised to run workloads on your controlplane and etcd nodes, there are occasionally scenarios when this is necessary. A few common examples are virus scanning, monitoring, and log collection workloads.

Pre-requisites

• A Rancher Kubernetes Engine (RKE) or Rancher v2.x provisioned Kubernetes cluster

Steps

Both the controlplane and etcd nodes, which are not additionally designated the worker role, have taints. When RKE or Rancher provisions these nodes, it adds these taints automatically. Workloads that need to run on these nodes require tolerations for these taints. For Rancher managed clusters, you can see these taints within the Rancher UI on the cluster node view. The following kubectl command will also list the taints for each node.

$ kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
NAME           TAINTS
ip-10-0-2-10   [map[effect:NoExecute key:node-role.kubernetes.io/etcd value:true]]
ip-10-0-2-11   [map[effect:NoSchedule key:node-role.kubernetes.io/controlplane value:true]]
ip-10-0-2-12   <none>

Per this output, each etcd node has the NoExecute taint node-role.kubernetes.io/etcd=true and each controlplane node has the NoSchedule taint node-role.kubernetes.io/controlplane=true.

The Rancher UI does not have fields for adding tolerations, so you must specify the tolerations directly in the workload's YAML manifest. You can use the Import YAML button to deploy your workload, and make sure to add the following tolerations block in your manifest:

spec:
...
  template:
...
    spec:
...
      tolerations:
      - operator: Exists
...

If you have an existing workload, you can also select the View/Edit YAML option for the workload and apply the above change. This toleration will allow you to run the workload on any nodes with taints, so use with caution. If you are using Helm charts, you can also specify the same YAML in your Helm chart.

Further Reading

For more information on how taints and tolerations work in Kubernetes, see: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/