Resolving Pod Distribution Issues with Topology Spread Constraints

Rancher 2.x

RKE2

When deploying applications with multiple replicas, the pods are not distributed evenly across the cluster's nodes. This leads to some nodes becoming heavily loaded while others remain underutilised, which can cause performance issues and a single point of failure. 

To ensure pods are spread out to balance the workload and improve high availability,  consider using topology-spread-constraints. This feature gives the Kubernetes scheduler a rule to follow, instructing it to distribute pods as evenly as possible across different parts of your cluster, such as nodes or availability zones.

This example shows a simple deployment for an application with three replicas that will be evenly spread across different nodes using topology-spread-constraints.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-app
  template:
    metadata:
      labels:
        app: web-app
    spec:
      containers:
      - name: web-app-container
        image: nginx:latest
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: "kubernetes.io/hostname"
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: web-app

• maxSkew: 1: This is the core rule. It means the difference in the number of pods between any two nodes in the cluster can be no greater than 1. This ensures a balanced spread.

• topologyKey: "kubernetes.io/hostname": This tells the scheduler that the "topology domain" is each individual node. The scheduler will count the number of web-app pods on each node to determine if the maxSkew is met.

• whenUnsatisfiable: DoNotScheduleThis tells the scheduler that if it can't find a node that satisfies the rule, it should not schedule the pod and leave it in a Pending state.
  OR
  if you select whenUnsatisfiable: ScheduleAnyway, the scheduler gives higher precedence to topologies that would help reduce the skew.

• labelSelector: This is how the scheduler identifies the group of pods it needs to balance. In this case, it looks for any pods with the label app: web-app

By default, the Kubernetes scheduler's primary goal is to find a node with enough resources to fit a new pod. It doesn't inherently prioritise an even distribution across the cluster. This can result in a "bin-packing" effect, where the scheduler places pods on the first nodes it finds that can accommodate them, leading to an unbalanced workload. topology-spread-constraints explicitly counter this default behaviour by introducing a balancing rule.

https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/