This article aims to provide a number of checks that can be evaluated to ensure best practices are in place when planning, building or preparing a Rancher 2.x and Kubernetes environment.

1. Architecture

1.1 Nodes

Understanding workload resource needs in downstream clusters upfront can help choose an appropriate node configuration; some nodes may need different configurations; however, all nodes of the same role are generally configured the same.

for n in $(kubectl get nodes -l node-role.kubernetes.io/master=true --no-headers | cut -d " " -f1)
  do
    kubectl get nodes --field-selector metadata.name=${n} --no-headers
    kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=${n}; echo
done

for n in $(kubectl get nodes -l node-role.kubernetes.io/etcd=true --no-headers | cut -d " " -f1)
  do
    kubectl get nodes --field-selector metadata.name=${n} --no-headers
    kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=${n}; echo
done

for n in $(kubectl get nodes -l node-role.kubernetes.io/controlplane=true --no-headers | cut -d " " -f1)
  do
    kubectl get nodes --field-selector metadata.name=${n} --no-headers
    kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=${n}; echo
done

1.4 Load balancer

To provide a consistent endpoint for the Rancher management cluster, a load balancer is highly recommended to ensure the Rancher agents, UI, and API connectivity can effectively reach the Rancher deployment.

1.5 Proximity and latency

For performance reasons, it is recommended to avoid spreading cluster nodes over long distances and unreliable networks. For example, nodes could be in separate AZs in the same region, the same datacenter, or separate nearby data centres.

This is particularly important for etcd nodes which are sensitive to network latency, the RTT between etcd nodes in the cluster will determine the minimum time to complete a commit .

1.6 Datastore

It is important to ensure that the chosen datastore is capable of handling requests inline with the workload of the cluster.

Allocation of resources, storage performance, and tuning of the datastore may be needed over time, this could be due to an increase in churn in a cluster, downstream clusters growing in size, or the number of downstream clusters Rancher is managing increases.

1.7 CIDR selection

The cluster and service CIDRs cannot be changed once a cluster is provisioned.

For this reason, it is important to future proof by changing the ranges to avoid routing overlaps with other areas of the network and potential cluster IP exhaustion if the defaults are not suitable.

1.8 Authorized cluster endpoint

At times connecting directly to a downstream cluster may be desired, this could be to reduce latency, avoid interruption if Rancher is unavailable, or that a high frequency of external API calls occur, for example, external monitoring, or a CI/CD pipeline.

2. Best Practices

2.1 Installing Rancher

It is highly encouraged to install Rancher on a Kubernetes cluster in an HA configuration .

If starting with small resource requirements, at the very minimum always install on a Kubernetes cluster with a single node, this provides a future path to adding nodes at a later date.

The design of the single node Docker install is for short-lived testing environments, migration from a Docker to a Kubernetes install is not possible.

2.2 Rancher Resources

The minimum resource requirements for nodes in the Rancher management cluster need to scale to match the number of downstream clusters and nodes; this may change over time and need reviewing as changes occur in the environment.

2.3 Chart options

When installing the Rancher helm chart, the default options may not always be the best fit for specific environments.

2.4 Supported versions

When choosing or maintaining the components for Rancher and Kubernetes clusters the product lifecycle and support matrix can be used to ensure the versions and OS configurations are certified and maintained.

2.5 Recurring snapshots and backups

It is important to configure snapshots on a recurring schedule and store these externally to the cluster for disaster recovery.

2.6 Provisioning

Provisioning nodes and resources for Rancher and downstream clusters in a repeatable and automated way will greatly improve the supportability of Rancher and Kubernetes. This allows nodes to be replaced in a cluster easily, and new clusters created in a consistent way.

2.7 Managing node lifecycle

When making significant planned changes it is important to drain nodes that are being affected to avoid disrupting in-flight connections, such as restarting Docker, patching, shutting down or removing nodes.

For example, the kube-proxy component manages iptables rules on nodes to manage service endpoints, if a node is suddenly shutdown, stale endpoints and orphaned pods can be left in place for a period of time causing connectivity issues.

In some cases during an unplanned issue, draining can be automated, such as when a node may be terminated, restarted, or shutdown.

3. Operating Kubernetes

3.1 Capacity planning and Monitoring

It is recommended to measure resource usage of all clusters by enabling monitoring in Rancher, or your chosen solution. It is recommended to alert on resource thresholds and events in the cluster.

On supported platforms, using Cluster Autoscaler can be used to ensure the number of nodes is right-sized for the pod workload. Combining this with Horizontal Pod Autoscaler provides both application and infrastructure scaling capabilities.

3.2 Probes

In the defence against service and pod related failures, liveness and readiness probes are very useful; these can be in the form of HTTP requests, commands, or TCP connections.

3.3 Resources

Assigning resource requests to pods allows the kube-scheduler to make more informed placement decisions, avoiding the "bin packing" of pods onto nodes and resource contention.

Limits also offer value in the form of a safety net against pods consuming an undesired amount of resources.

In addition to defining requests and limits for pods, it can also be useful to reserve capacity on nodes to prevent allocating resources that may be consumed by the kubelet and other system daemons, like Docker.

3.4 OS Limits

Containerized applications can consume high amounts of OS resources, such as open files, connections, processes, filesystem space and inodes.

Often the defaults are adequate; however, establishing a standardized image for all nodes can help establish a baseline for all configuration and tuning.

3.5 Log rotation

To prevent large log files from accumulating, and apply a desired retention period it is recommended to rotate OS, pod log files, and configure an external log service to stream logs off the nodes for a longer-term lifecycle and easier searching.

3.6 DNS scalability

DNS is a critical service running within the cluster. DNS queries are distributed throughout the cluster, where the availability depends on the accessibility of the CoreDNS pods in the service.

The Nodelocal DNS cache is a redesign on the architecture and is recommended for clusters that may experience high DNS workload or issues.