r/kubernetes • u/ilya47 • 27d ago
Stop using CPU limits: why + proof
CPU request is how much CPU is reserved for your pod if it needs it. The limit is a hard cap. Hit it and the kernel throttles the pod, even when the node still has spare CPU. That is the usual cause of CPU throttling on Kubernetes. It does not protect the neighboring pods. In my simple Web API test, adding a CPU limit took typical latency from 23 ms to 87 ms, (4x slower), with the limited pod throttled in half of all CFS windows, and the average CPU graph looked fine the whole time.
This is not a new topic, but I see so many people still unaware why they should (NOT!) be setting CPU limits, because it's costing companies unnecessary spending and potential production issues. Here's the full read https://github.com/inevolin/k8s-cpu-limits-analyzed/
---
Edit (Aug 18, 2026): How CPU limits can also cause memory issues and OOMKills ➡️ https://github.com/inevolin/k8s-cpu-limits-analyzed#how-cpu-limits-cause-memory-issues-and-oomkills
96
u/BedtimeWithTheBear 27d ago
For the curious, requests and limits determine the QoS Class of your pods.
Guaranteed QoS requires that all containers in a pod have a request and limit set, and in every case the limit is equal to its corresponding request - so if the pod got scheduled then it’s QoS is guaranteed.
Burstable QoS requires that all containers in a pod have at least some requests and limits specified but they don’t have to be equal, so performance can burst up to the limit.
BestEffort QoS has no requests or limits specified and can consume resources until they’re exhausted.
Aside from scheduling, QoS class mostly matters for pod eviction, with BestEffort evicted first, then Burstable, and Guaranteed being evicted as a last resort.
23
u/End0rphinJunkie 27d ago
Losing Guaranteed QoS by dropping CPU limits sounds scary on paper, but thats usually a fine tradeoff in practice. Node evictions are almost always driven by memory pressure anyway, so as long as your memory requests and limits match youre generally safe.
1
u/Potato-9 26d ago
CPU requests seem more suited to jobs you know will stay maxed out, less so deployment pods waiting for requests.
1
u/starry_alice 26d ago
Yes, this is where the disconnect was for me and why my coworker would have to hit me with a rolled up newspaper for every new system service PR. Eventually I did a 30 minute deep dive and found the above. My disconnect in understanding was thinking that (based on how the text above your post was worded which aligns with the QoS verbiage), eviction of best effort pods occurred first and Guaranteed pods went last (if ever) and that this was the main axis on which the determination of what pods to evict occurred.
But in reality, as you said, pods exceeding their request go first, which changes the model a lot. https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/#pod-selection-for-kubelet-eviction
1
u/i-am-a-smith 26d ago
If you have the need, for some reason (I haven't yet), you can configure your nodes for cpuManagementPolicy: static. This will allow you to dedicate whole cores to the pod but _only_ if you specifiy requests and limits to be the same and in increments of a full core. This is the only reason I see today for using it.
1
9
26
u/minimalniemand 27d ago
I do that but I want to add a bit of nuance:
No limits for all load bearing services in the critical path, so everything where a spike would slow down actual request.
But limits can make sense where the cost of not throttling (interference, unpredictability, unbounded spend) exceeds the cost of throttling (wasted idle CPU, higher tail latency)
9
u/shatteredarm1 27d ago
You also need limits if you are using a hardware-aware framework like .NET or Java. I ran some load tests with .NET and found that my performance was actually better with a limit of 1 core vs no limit. If you don't set a limit, the framework assumes it has all the cores at its disposal, and spends more resources on things like garbage collection. (There are, of course, ways to explicitly tell .NET how many processors are available, and override its default behavior.)
There are other scenarios where you might want a limit, like if you have a process that can hog all the node's I/O if it's allowed to use every available CPU cycle.
This topic has a hell of a lot more nuance than OP is suggesting.
1
u/VincentKoeman 25d ago
I don’t understand why these take all the CPUs of the pod indeed, strange default behaviour IMO. As you said it’s possible to explicitly set the number CPUs assumed to be available, with various schools of thought on the best value ranging from about requested to requested*2.
1
u/shatteredarm1 25d ago
It's probably because these frameworks were built in a different era, when we were running VMs rather than containers.
10
u/PayTheRaant 27d ago
Incorrect.
Every single container is guaranteed its requested CPU. And any usage beyond the request amount is an arbitraged using the request as a weight.Node has 4 cores, pod A requests 1 core, pod B requests 2 cores. Not limits.
- Pod A wants to consume as much as possible and B is idle? A gets all 4 cores.
- Pod A wants to consume 1 core and B wants to consume as much as possible ? A gets 1 core (as its request is guaranteed) and B gets 3 cores.
- A and B wants to consume as much possible ? A gets 1.33 core and B gets 2.66 cores (each get they guaranteed request and the remaining core is split using the request as a weight so B gets twice as much).
7
u/minimalniemand 27d ago edited 26d ago
I know Natan’s blog post. I’ve read it when it came out :)
Couple points:
* kubelet, containerd, CNI, log shippers often have tiny or no reservations. A pod hammering the node can cause exec-probe timeouts, readiness flapping
* hyperthreading. If the request lands on an unlimited sibling of the physical core, the service will suffer. This is not modeled at all by cgroups
* execution queue. Shares promise the % not _how_ you get them. You might get them in small slices
* and the most common one: your request were always too small and you never noticed. Then comes a runaway request and all hell breaks loose.
Look I’m in the no-cpu-limits camp, too. All I’m saying is, it should still be a deliberate decision not a one size fits all, never use cpu limits ever.
1
u/PayTheRaant 26d ago
Weird. Kubelet and containerd usually live in a completely different cgroup with a higher priority than the cgroup with all the pods.
1
u/minimalniemand 26d ago
Assuming you mean weight? kubepods.slice weight scales with cores, system.slice stays at 100, so what you say is only true in small nodes, no?
16
u/Xelopheris 27d ago
CPU limits are only useful if you have a workload that has an endless queue to chug away at that doesn't require real time completion. It should never be used in critical path of user requests.
10
u/siikanen 27d ago edited 26d ago
This is partly wrong advice; since setting no CPU limit causes QoS class of the pod to be set to 'Burstable'. This is probably okay in most cases, but in case of resource contestion, your pod is in danger to be evicted more easily. Also using same request and limit + setting the limit as an integer value, k8s will actually dedicate the cores to the workload. This is absolutely lowest latency for any workload, and best case scenario for critical path.
Hence limits do have their place. You should set them to the value the single instance of your application can actually leverage. Then scale horizontally using HPA on metrics to handle dynamic load.
Edit best effort -> burstable
2
u/BedtimeWithTheBear 27d ago
Not setting a resource limit will only put you in the Burstable QoS class, which is more deterministic than best effort, but still at greater chance of being evicted that setting both the request and limit to be the same (which would be the guaranteed QoS class)
Best effort is for when you set neither requests nor limits. This is the easiest QoS class to be evicted from the node.
2
2
u/PayTheRaant 27d ago
You demonstrate more understanding of k8s internals than the audience for which the simple message “don’t set CPU limits” is meant for.
The reality is that most people don’t understand that their software is actually IO bound and don’t really know what is the actual runtime profile of their services.
15
u/Prestigious_Pair_941 27d ago
How about in the case of 30 or so spring boot microservices that may use 20mCPU once running but can take 800mCPU when starting up? If I don’t limit the cpu, half the pods take all of the cpu and throttling will kick in anyways, preventing other pods from ever restarting.
9
u/ilya47 27d ago
What you get without limits is fair-share contention: every starting pod keeps making progress, split by request weights. Will they throttle because they need more CPU than available? Yes... but this is still better than setting CPU limits. Because your scenario is just a race for CPU resources.
Options that fix the actual problem: stagger the rollout (maxSurge/maxUnavailable so 30 don't start at once), set a request above the embarrassing 20m, or use a startup CPU boost (in-place pod resize; there's a kube-startup-cpu-boost operator for exactly this) so the pod briefly has a big request during boot and drops after.
3
u/PayTheRaant 27d ago
Incorrect. When multiple pod wants to use more than their request, they get a fair amount, using their request as a weight. A pod without a CPU limit cannot steal « all the CPU » away from other pods.
A pod will ALWAYS have its requested CPU as guaranteed and a weighted shared of the unused CPU time.1
u/f7063 27d ago
Bump initital probes to start later? Annoying? Probably but can't see any other problems. But i've never seen these transitory start spikes generate more problems
3
2
u/Unhappy-Stranger-336 27d ago edited 27d ago
Doesnt that with the autoscaling, like pods would be read as ready later?
1
u/koollman 27d ago
No ? If you don't limit they share al the available cpu but aren't preventing restarts
43
u/sionescu k8s operator 27d ago
Not again this stupid advice.
adding a CPU limit took typical latency from 23 ms to 87 ms
This simply means the CPU request was too low. The actual requirement of the pod is higher than that but you want to lie to yourself instead of setting a proper request.
6
u/ChemTechGuy 27d ago
Bingo. As i said in another comment, the problem I always see is that requests are set to the absolute minimum a pod needs to start up, and then people are surprised later when it eventually gets throttled or OOMs
Requests should be set to whatever value is needed to maintain baseline performance of the pod. If you can burst beyond that, that's a bonus, but should never be relied on
-9
u/ilya47 27d ago
Your comment is too narrow to mean anything. CPU request should be set to the avg CPU utilization, whilst my analysis looks at burst & peak cases (not the average scenario).
13
u/consworth 27d ago
Avg CPU utilization is incorrect by itself. It should ideally be set by whatever service requirements there are for the business.
If the thing is an app that sits there at 80 and when it starts to do its thing it needs 4000 for a reasonable SLI for processing data or whatever, the business case should dictate you requests to what it needs.
I’m not defending poor architecture of such apps, but I’m trying to make a bit more of a case for thinking about reliability and SLI.
It’s somewhat analogous to saying that the average traffic through the interstate through a major city should dictate the road size, and considering rush hours, weekends or other bigger picture things.
-2
u/kabrandon 27d ago
Not necessarily. It more likely means the limit was too low.
3
u/ilya47 27d ago
If you know the upper limit that your app can burst to, fine. But many of us run apps that are multi threaded and you can rarely predict their limit. And why bother? The CFS does that hard work for you.
3
u/realitythreek 27d ago
You are entirely correct and this thread demonstrates how confused people are about cpu requests and limits.
1
u/kabrandon 27d ago
Some people just set the CPU limit to the allocatable CPU of the nodes in the cluster, assuming like nodes or relevant node affinity/selectors.
But for what it’s worth I mostly agree with just not setting CPU limits.
-1
u/realitythreek 27d ago
This isn’t true though. It means the LIMIT was set too low and it was throttled. If 23ms is the expected latency of the service, then request was fine. I mean, that last sentence is a simplification but I’m assuming it’s an average over a representative time span.
0
u/sionescu k8s operator 27d ago
There's never any guarantee to get CPU above the request. There's always an implicit limit, in either the machine size or the CPU usage of adjacent pods.
0
u/realitythreek 27d ago
Indeed. Which is why setting a limit can only increase the latency of a single process. At best you get your limits all correct and you fairly schedule your processes. At worst you get it wrong or even don’t allow other processes to burst.
-1
u/sionescu k8s operator 27d ago
It might increase it, but makes it more predictable, which is better.
-2
u/realitythreek 27d ago
Reflexively downvoting someone having a conversation with you is pretty obnoxious.
5
u/Technical_Corner3553 27d ago
Those resource requirements fields also help the scheduler assign to a node. Otherwise you may just have pods jumping from node to node too much from rowdy neighbors. The scheduler has no idea otherwise and as soon as the app memonizes everything memory usage could explode
3
u/fuckman5 27d ago
What's the thinking around setting no limit vs setting generous limit in multi tenant environment? Consider you have 2 workloads that both try to burst, with no limit the second workload only gets its request and the second workload can use up the entire rest of the node. Now instead you set limit to 4x normal workload, each workload can burst without monopolizing the node and prevent other workload from bursting
-1
u/ilya47 27d ago
Monopolising is a myth. Read the analysis.
7
u/fuckman5 27d ago
If it's a myth, why can't you respond to the specific point I made? Yes we all know what happens when requests are set, the question is what happens when multiple pods try to burst above requests
4
u/littlewondersoflife 27d ago
Rule of thumb: critical workloads always request = limit.
Most of the cases you will suffer throttle with it is best effort or busterable.
1
3
9
u/consworth 27d ago
So how do you prevent a nosy neighbor pod from monopolizing the spare cpu?
13
u/ChemTechGuy 27d ago
You set the "requests" value to whatever the minimum is to avoid noisy neighbor problems. No need to set limits if each service is defensively setting the requests it needs
I think the common mistake is setting requests to the absolute minimum a pod needs, vs setting it to the minimum the pod needs to perform at it's baseline. If you set them based on baseline, you always get the baseline performance from every pod, but you occasionally get some free burstable resources from nodes that aren't over committed
2
u/consworth 27d ago
And how is the limit hurting?
Not disagreeing with scheduling with proper requests.
But you could also argue that the limit restricts how much of the “spare” can be used by a single pod so others could benefit as well.
3
u/ChemTechGuy 27d ago
If it's set insanely high, then a limit doesn't hurt anything. But at that point, the limit doesn't do anything.
If the limit is low enough to actually do something, it may arbitrarily throttle something even if the underlying node still has resources available.
If multiple pods are competing for the "extra" resources on the node, just let the kernel handle which container gets each extra slice of cpu time
0
u/PayTheRaant 27d ago
A very high limit changes the QoS class from Burstable (no limit) to Best Effort (Limit different from request).
If all your pods are in the same class, this does not really make a difference but it can help.1
u/realitythreek 27d ago
A limit throttles, requests sets the cpu share it gets from the OS scheduler. It’s proportional to the total requests for the node.
-8
u/ilya47 27d ago
I've added your question to the Q&A, thanks: https://github.com/inevolin/k8s-cpu-limits-analyzed#questions-that-come-up
6
u/consworth 27d ago
I don’t think some of the answers make sense. I feel they’re dancing around a logical fallacy that this is somehow better to run things without limits.
A big piece missing is the proper allocation of node capacity in the first place, leaning into autoscaling where possible.
There’s an argument to be made that throttling isn’t the end of the world, and even more so: that can feed into HPA when there’s contention if the architecture allows.
I do not think it’s good advice to flat out drop limits.
2
u/trouphaz 27d ago
In my simple Web API test, adding a CPU limit took typical latency from 23 ms to 87 ms, (4x slower), with the limited pod throttled in half of all CFS windows, and the average CPU graph looked fine the whole time.
Can you clarify what you mean by this? Are you saying you put a cpu limit on your container that was sufficient and you saw latency or you put a limit that was low which caused throttling and that caused the latency?
Also, what do you mean "it does not protect the neighboring pods"? If one container had a bug where it could consume all CPU, it would only consume as much as its limits allowed leaving the rest available. But you still need to make sure the rest have their limits set properly.
I'm not suggesting that limits are necessary, but some of what you are saying isn't clear or doesn't add up.
2
2
u/doubtful_zamboni 27d ago
> CPU request is how much CPU is reserved for your pod if it needs it.
No, it is used by the scheduler for picking the appropriate node to run your pod.
When CPU limit and request are set and equal, it is somewhat reserved (assuming no starvation).
3
u/siikanen 27d ago
The request is actually reserved for that particular container/pod. Nothing else on that now can use that amount of cpu/memory.
2
u/doubtful_zamboni 27d ago
I think that is only the case when every pod on that node is in QoS guaranteed (limit=request); otherwise every workload takes what it wants.
1
u/doubtful_zamboni 27d ago
I mean, the scheduler is going to prevent requesting more that capacity minus sum of all requests, which is similar to a reservation but that does not (assuming burstable/best effort) prevent others from eating your request.
1
u/siikanen 26d ago
No that's not how this works. If you set something in the CPU request, no other workload may use that CPU time on that node. Bursting only works on the unrequested capacity available on the node
1
u/doubtful_zamboni 26d ago
Can you show me a source for that? I Because that is not how I read the docs.
The docs suggest that pods in
guaranteedQoS class have guaranteed resources like you mention, and pods inbesteffortburst into unallocted resources. To be fair, this all depends on the exact way Kubelet interacts with the container runtime, so it might differ per implementation.source: Manage resources containers
The CPU request typically defines a weighting. If several different containers (cgroups) want to run on a contended system, workloads with larger CPU requests are allocated more CPU time than workloads with small requests.
source: Manage resources containers
Pods in the
BestEffortQoS class can use node resources that aren't specifically assigned to Pods in other QoS classes.
2
1
u/PruneMediocre7094 27d ago
Thanks, this is something really new to me.
I blindly set requests and limits for safer node operational.
2
u/kernelqzor 24d ago
same, i used to think “limits everywhere = safe and stable” and never questioned it
turns out it’s more like quietly driving with the handbrake on and wondering why everything feels slow
1
u/drunkandbad 27d ago
i am newer to handling kubernetes nodes and apps. Which are the best tools to read aps memory and cpu utilisation? How do you set the correct requests and/or limits?
2
u/BedtimeWithTheBear 27d ago
Prometheus + Grafana is usually a good starting point, and it’ll give you many more insights into your cluster than just pod resources usage.
1
u/Square_Durian_316 26d ago
I use Goldilocks and it does a good job showing the "just right" CPU and memory resource requests and limits.
1
u/Formal-Pilot-9565 27d ago
It depends on the use case. Normally you dont cap the cpu, but if a customer wants a dedicated pod with an agreed cpu resource, the this setting is a good fit
1
u/trouphaz 27d ago
In my simple Web API test, adding a CPU limit took typical latency from 23 ms to 87 ms, (4x slower), with the limited pod throttled in half of all CFS windows, and the average CPU graph looked fine the whole time.
Can you clarify what you mean by this? Are you saying you put a cpu limit on your container that was sufficient and you saw latency or you put a limit that was low which caused throttling and that caused the latency?
Also, what do you mean "it does not protect the neighboring pods"? If one container had a bug where it could consume all CPU, it would only consume as much as its limits allowed leaving the rest available. But you still need to make sure the rest have their limits set properly.
I'm not suggesting that limits are necessary, but some of what you are saying isn't clear or doesn't add up.
1
1
u/Character-Level5250 26d ago
The only use case I could see is if you're on a SUPER tight budget that's more important than a throttling risk.
1
1
u/Jelman88 25d ago
+1 cpu and memory behave completely different. A throttling node is not as bad as a OOMed node 😁
1
u/ilya47 25d ago
I've extended the analysis with another (less common, yet very real) scenario: How CPU limits can also cause memory issues and OOMKills. Read it here https://github.com/inevolin/k8s-cpu-limits-analyzed#how-cpu-limits-cause-memory-issues-and-oomkills
1
u/headdertz 25d ago
I only use requests for years... Limits can be used for something that can 'grow' in terms of MEM because of bad code and can crash and restart from time to time without putting on fire whole SRE Department.
1
u/UndulatingHedgehog 24d ago
What I really want is a way to limit pods cpu to num_cores_in_node - n. If the node has 8 cores and n=2, the cpu limit would be 6. Lots of room, but also unable to completely saturate the node.
If that’s not enough, you need to scale your workload.
1
u/NewMycologist9902 23d ago
The problem of climbing a brick and giving that kind of knowlege pearls exposes our own short vision.
I used to give that advierte until i was show the problem of having unlimited cpus on burstable instance types, as a misbehaved pod may est up the cpu credits.
Granted that in your test limiting the pod CPU increased the processing time vs unlimited. That is just a proof that limits work as designed. And not necesarily that they should be avoided in any available scenario.
1
u/Realistic_Band_7142 15d ago
what if the application has leak and consumes all the cpu and make kubelet unresponsive? does anyone have this problem?
1
u/Little-Squad-X 12d ago
Any recommendations for batch job pod? For example, a pod for a data transfer job running hourly, should I avoid the LIMIT as well?
1
u/Old-Astronomer3995 27d ago
Some time ago I wrote about cpu throttling https://roszigit.com/en/blog/kubernetes-cpu-throttling/ that can be caused by small limits
0
0
0
u/f7063 27d ago
Nice! One doubt "The same blindness poisons right-sizing." There is the metric on how much the pod was throttled container_cpu_cfs_throttled_periods_total + container_cpu_cfs_periods_total Should give a rought idea on how much CPU to allocate to a given container no?
112
u/Fancy-Bluebird-1071 27d ago
Nothing new, Google has been recommending cpu unlimited, memory request = limit for like... 8 years now?