This article aims to provide a number of checks that can be evaluated to ensure best practices are in place when planning, building or preparing a Rancher 2.x and Kubernetes environment.

1. Architecture

1.1 Nodes

Understanding workload resource needs in downstream clusters upfront can help choose an appropriate node configuration; some nodes may need different configurations; however, all nodes of the same role are generally configured the same.

for n in $(kubectl get nodes -l node-role.kubernetes.io/master=true --no-headers | cut -d " " -f1)
  do
    kubectl get nodes --field-selector metadata.name=${n} --no-headers
    kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=${n}; echo
done

for n in $(kubectl get nodes -l node-role.kubernetes.io/etcd=true --no-headers | cut -d " " -f1)
  do
    kubectl get nodes --field-selector metadata.name=${n} --no-headers
    kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=${n}; echo
done

for n in $(kubectl get nodes -l node-role.kubernetes.io/controlplane=true --no-headers | cut -d " " -f1)
  do
    kubectl get nodes --field-selector metadata.name=${n} --no-headers
    kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=${n}; echo
done

1.4 Load balancer

To provide a consistent endpoint for the Rancher management cluster, a load balancer is highly recommended to ensure the Rancher agents, UI, and API connectivity can effectively reach the Rancher deployment.

1.5 Proximity and latency

For performance reasons, it is recommended to avoid spreading cluster nodes over long distances and unreliable networks. For example, nodes could be in separate AZs in the same region, the same datacenter, or separate nearby data centres.

This is particularly important for etcd nodes which are sensitive to network latency, the RTT between etcd nodes in the cluster will determine the minimum time to complete a commit.

1.6 Datastore

It is important to ensure that the chosen datastore is capable of handling requests inline with the workload of the cluster.

Allocation of resources, storage performance, and tuning of the datastore may be needed over time, this could be due to an increase in churn in a cluster, downstream clusters growing in size, or the number of downstream clusters Rancher is managing increases.

docker exec -e ETCDCTL_ENDPOINTS=$(docker exec etcd /bin/sh -c "etcdctl member list | cut -d, -f5 | sed -e 's/ //g' | paste -sd ','") etcd etcdctl check perf

export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml
etcdcontainer=$(/var/lib/rancher/rke2/bin/crictl ps --label io.kubernetes.container.name=etcd --quiet)
crictl exec $etcdcontainer sh -c "ETCDCTL_CACERT='/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/rke2/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/rke2/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl check perf

1.7 CIDR selection

The cluster, node, and service CIDRs cannot be changed once a cluster is provisioned.

For this reason, it is important to future proof by reviewing the ranges to avoid routing overlaps with other areas of the network, and potential cluster IP exhaustion if the defaults are not suitable.

1.8 Authorized cluster endpoint

At times connecting directly to a downstream cluster may be desired, this could be to reduce latency, avoid interruption if Rancher is unavailable, or avoid proxying the requests when a high frequency of API calls are expected - for example, external monitoring, automation or a CI/CD pipeline.

2. Best Practices

2.1 Installing Rancher

It is highly encouraged to install Rancher on a Kubernetes cluster in an HA configuration.

When starting with a single node for the Rancher management (local) cluster, at the minimum it is highly recommended to install on a single node Kubernetes cluster to improve configuration and management of the Rancher environment.

The intended use case of the single node Docker install is for short-lived testing environments, migration from a Docker to a Kubernetes install is not possible and will require migrating Rancher to the new cluster using the Backup Operator.

2.2 Rancher Resources

The minimum resource requirements for nodes in the Rancher management (local) cluster need to scale to match the number of downstream clusters and nodes, this may change over time and need reviewing as changes occur in the environment.

2.3 Chart options

When installing the Rancher helm chart, the default options may not always be the best fit for specific environments.

2.4 Supported versions

When choosing or maintaining the components for Rancher and Kubernetes clusters the product lifecycle and support matrix can be used to ensure the versions and OS configurations are certified and maintained.

2.5 Recurring snapshots and backups

It is important to configure snapshots on a recurring schedule and store these externally to the cluster for disaster recovery.

2.6 Provisioning

Provisioning clusters, nodes and workloads for Rancher and downstream clusters in a repeatable and automated way can improve the supportability of Rancher and Kubernetes. When configuration is stored in source control it can also assist with auditing changes.

2.7 Managing node lifecycle

When making significant planned changes it is important to drain nodes that are being affected to avoid disrupting in-flight connections, such as restarting Docker, patching, shutting down or removing nodes.

For example, the kube-proxy component manages iptables rules on nodes to manage service endpoints, if a node is suddenly shutdown, stale endpoints and orphaned pods can be left in place for a period of time causing connectivity issues.

In some cases during an unplanned issue, draining can be automated, such as when a node may be terminated, restarted, or shutdown.

3. Operating Kubernetes

3.1 Capacity planning and Monitoring

It is recommended to measure resource usage of all clusters by enabling monitoring in Rancher, or your chosen solution. It is recommended to alert on critical resource thresholds and events in the cluster.

On supported platforms, Cluster Autoscaler can be used to ensure the number of nodes is right-sized for shceduled workloads. Combining this with Horizontal Pod Autoscaler provides both application and infrastructure scaling capabilities.

3.2 Probes

In the defence against service and pod related failures, liveness and readiness probes are very useful, these can be in the form of HTTP requests, commands, or TCP connections.

3.3 Resources

Assigning resource requests to pods allows the kube-scheduler to make more informed placement decisions, avoiding the "bin packing" of pods onto nodes and resource contention. For CPU requests, this also allocates a share of CPU time based on the request when the node is under contention.

Limits also offer value in the form of a safety net against pods consuming an undesired amount of resources. One caveat specific to CPU limits, these can introduce CPU throttling when over used.

Additionally, it can also be useful to reserve capacity on nodes to prevent allocating resources that may be consumed by the kubelet and other system daemons, like Docker.

3.4 OS Limits

Containerized applications can consume high amounts of OS resources, such as open files, connections, processes, filesystem space and inodes.

Often the defaults are adequate; however, establishing a standardized image for all nodes can help establish a baseline for all configuration and tuning.

3.5 Log rotation

To prevent large log files from accumulating, it is recommended to rotate OS and container log files. Optionally an external log service can be used to stream logs off the nodes for a longer-term lifecycle and easier searching.

kubelet-arg:
  - container-log-max-files=4
  - container-log-max-size=50Mi

3.6 DNS scalability

DNS is a critical service running within the cluster, as CoreDNS pods are scheduled throughout the cluster, the service availability depends on the accessibility of all CoreDNS pods in the service.

This is where Nodelocal DNS cache is recommended for clusters that may service a high amount of DNS requests, or clusters very senstive to DNS issues.