scheduling - Kubernetes Pod anti-affinity - evenly spread pods based on a label?

Question

We are finding that our Kubernetes cluster tends to have hot-spots where certain nodes get far more instances of our apps than other nodes.

In this case, we are deploying lots of instances of Apache Airflow, and some nodes have 3x more web or scheduler components than others.

Is it possible to use anti-affinity rules to force a more even spread of pods across the cluster?

E.g. "prefer the node with the least pods of label component=airflow-web?"

If anti-affinity does not work, are there other mechanisms we should be looking into as well?

Answer 1

Have you tried configuring the kube-scheduler?

kube-scheduler selects a node for the pod in a 2-step operation:

• Filtering: finds the set of Nodes where it's feasible to schedule the Pod.
• Scoring: ranks the remaining nodes to choose the most suitable Pod placement.

Scheduling Policies: can be used to specify the predicates and priorities that the kube-scheduler runs to filter and score nodes.

kube-scheduler --policy-config-file <filename>

• Sample config file

One of the priorities for your scenario is:

• BalancedResourceAllocation: Favors nodes with balanced resource usage.