In Kubernetes, are resource quotas a good way to throttle how much CPU and memory is allowed for running jobs at a given time?

Question

Suppose I have an API that allows users to create jobs (V1Jobs) in Kubernetes. What I would like is for the user to be able to submit as many jobs as they want without a failed response from the API but to have Kubernetes queue/throttle the jobs until there are enough available resources in the given namespace. For example, suppose I create a resource quota and specify a limit of 1cpu and 1Gi memory. Then suppose a user submits 100 1cpu/1Gi jobs. I'd like Kubernetes to process one at a time until they are complete. In other words running 100 jobs one at a time. Is creating the resource quota and letting the job-controller/scheduler handle the throttling the right way to go or would there be benefits to handle tracking the cluster usage externally (in an application) and only submit/create the V1Jobs to the API once there is capacity in the namespace?

question from:https://stackoverflow.com/questions/65927390/in-kubernetes-are-resource-quotas-a-good-way-to-throttle-how-much-cpu-and-memor

Answer 1

ResourceQuotas are a good start, limiting the amount of resources a that may be used within a namespace, or by resources matching an expression.

It would indeed prevent the scheduler from creating Pods that would exceed your quota limitations. The API would still accept new Job objects posted by clients. If you have N Jobs requesting 1 CPU/1G RAM, while your quota only allows for less than 2CPU/2G RAM to be used, you should see those jobs running sequentially.

Though it could still make sense to track how many pending/running jobs you have in your namespace, as this could show there are currently too many jobs to run with your current quota configuration. The kube-state-metrics exporter from Prometheus would gather the metrics you need for this, you'ld find sample dashboards in Grafana, alerting rules over there.

If there's a risk some containers would start without passing proper cpu or memory resources requests / limits, you could also look into LimitRanges, forcing some defaults.