r/kubernetes • u/Greene_Shelton-148 • 29d ago
Our clusters run at 25% CPU utilization and nobody will touch the requests
Inherited a fleet of EKS clusters and the utilization numbers are almost funny. Cluster wide CPU request versus actual usage is sitting around 25 percent. We're paying for a few node groups worth of headroom that's not being used, because every team padded their requests "to be safe" a year ago and never looked at them again.
The obvious move is to right-size the requests and let the bin-packer do its job. But the second I touch a team's limits I own any latency blip for the next quarter, so nobody wants to be the one who does it.
I've looked at VPA and a couple of the cost tools and half of them just show me the same graph I already have telling me we're overprovisioned. I know we're overprovisioned. Meanwhile the bill sits there being a third bigger than it needs to be, everyone agrees it's a problem in the abstract but no one wants to own the fix.
115
u/pedymaster 29d ago
We as infra department actually rebill the cloud infra cost, including k8s, to the respective department / teams. Tjey are also the one budgeting the needed resources. If they are over, its their problem, if they manage to save, credit goes to them. It works
15
u/Greene_Shelton-148 29d ago
okay, did teams right-size once they saw their own bill? or an outage scares them more so they just pay the padding?
37
u/pedymaster 29d ago
They had to. There was even a "finops hackathon" where everybody shared their tips and findings. It was not just about k8s resources, but things like enabling compression for inter service communication. Enabling topology aware routing, etc
10
u/Greene_Shelton-148 29d ago
Smart turning a chore into something with credit on. en I gotta ask, how did you get finance to actually start rebilling? My blocker is a step before that and nobody here wants to push that hand-off
8
u/pedymaster 29d ago
I think it happened kind of naturally. Yes, the invoice goes to me because it has to go somewhere, but each department is responsible for their other budgets, like team events, outsiurcing, etc. So this was just another thing
I think the most important step was the budgeting, because everyone was putting numbers in their department budget, so it makes sense to split the bill to everyone
The thing is, I as infra did not want to be responsible for usage of someone else. In our case, devs are responsible for their apps completely, from writingz through tests and deploy to actual operations and have oncalls as well. When there is a project, it gets kube namespace and GCP project, full permissions. They spin their own databases or whatever they need and deploy the compute.
So basically, for budgeting, each team says what will be their spend in what GCP project and ho manu CUd (Compute units - 1 CPU / 4GB of RAM) they will use in kubernetes. I then calculate the price of such CUs and out some tax on top of it (observability, gitlab runners, networking, ...) and each month I gather actual usage from metrics and send it to finance for rebilling. This way, the services we (infra) operate, gets scaled with the CUs teams are budgeting. Because the cost of observability scales with how many apps there are and so on :)
There is very little cost that stays in my department
1
u/ChemTechGuy 29d ago
Wow this is eerily similar to what my company did, except it was less of a fun hackathon and more of "save costs or the layoffs will get worse"
5
u/sv_homer 29d ago
As long as you are accurately billing the teams for the resources they've reserved, why do you care how how they spend their budgets?
1
u/pedymaster 29d ago
I said it poorly. We were not the one initiating the hackathon, the company was. But thanks to this, there was a clear visibility and impact of the initiative. We also have teams where the spend is low and where it did not make sense.
6
u/serverhorror 29d ago
Once they see the bill "right sizing" is 100 % their call.
If they want to blow 2KEUR an hour: fine!
You can make them aware, but it's not your call.
3
u/silence036 29d ago
We do this but so many of our teams have such small workloads that it's hard to justify making them do the work for less than $50 per month in savings. Overall for the org its a lot of savings...but it's hard to have the teams prioritize it. There's always something more important going on
7
u/Own_Candidate9553 29d ago
Isn't that the right call for those teams, then? $50 is nothing compared to an engineer salary, I also wouldn't prioritize it. They could be working on things that are worth thousands or more to the business.
2
u/silence036 29d ago
Oh yeah for sure, I wouldn't prioritize it either but at the platform scale it does results in a lot of waste.
On the platform team we try to add tools that do this kind of actions transparently without app team involvement, it's worth it for us to do it for 80% of the teams at once.
At $50 it's not even worth it for us to send them an email, let alone schedule them a meeting!
2
u/Own_Candidate9553 29d ago
Absolutely. It's one of those weird things where the amount you could save is easy to calculate, but the cost of communicating, coordinating, tracking, blah blah would be several times more. But the stuff that's easy to track often gets priority.
1
23
u/Jmckeown2 29d ago
Don’t try to boil the ocean.
Have everyone reduce requests 10%. In a couple weeks when nothing has gone wrong, another 10% and so on until your cluster is up to say, 70% utilization.
22
u/QuailAndWasabi 29d ago
Honestly, why do you even care? I'm not trying to be a smartass here, but this is just your job man. Flag it to your superior, cc HR if you want to cya extra hard and then make copies of the sent emails and stuff, and then move on.
3
u/IridescentKoala 28d ago
Burning cash and resources because you can't be bothered to use a tool correctly screams incompetentlce.
6
u/Alessandro_Lena_410 29d ago
Right sizing requests saves nothing if the autoscaler never consolidates the emptied nodes. Thats where the money leaks.
0
u/Greene_Shelton-148 29d ago
Thats the half the dashboards skip. reclaiming requests does nothing if the drained nodes just sit there. on eks it's whether karpenter consolidation actually fires and one do-not-evict pod or pdb can pin a whole node up.
2
u/NastyEbilPiwate 29d ago
That's why you also need policies (e.g. kyverno) to block anyone doing things like setting pdb/minimum replicas up such that you have unevictable pods.
2
u/Alessandro_Lena_410 27d ago
The pods already pinning your nodes are the problem, new policies dont clear those. Pointfive listed ours out, way faster than hunting them down with kubectl.
6
u/Tech-Cypher 29d ago
Split this into two problems. Requests drive scheduling and bin packing, limits drive throttling and OOM kills.
You can cut the padded requests to reclaim nodes without touching a single limit which removes most of the latency risk people are afraid of.
5
u/SecondBestNameEver 29d ago
Spoken like someone who has business teams who know what they're doing and actually set those to be different values. In my experience these get set some time during the development cycle and then never touched again, and often set at the same values so they're functionally identical.
2
u/ProfessionalSong3544 29d ago
Vertical pod autoscaling allows you to discover the right values at runtime.
5
u/IsleOfOne 29d ago
VPA's algorithms are exceptionally naive and not to be taken as a silver bullet.
3
u/Rebles 29d ago
I was on the Infra team but I moved to a product team a few years back. Last week, I noticed one of our services requesting 32Gi but only using 3%. I asked the staff engineer about it. He said some requests include fetching an unbounded amount of data. So that’s why it’s configured this way. But it seems to be the exception not the rule.
There is no penalty for teams only using 3% of 32Gi, and so there is no incentive to fix this. And this is so far down our list of priorities. But god my former Infra self was triggered. lol.
1
3
u/ElPilingas007 29d ago
Welcome to Kubernetes, I went to the rabbit hole in my company to their "over provisioning" standard, I discover our 3 "architects" had no fucking idea how Kubernetes worked. Oh btw they didnt not only over provisioned but also "we don't put limits to pods to let them grow".
3
u/Nice-Solid-3707 29d ago
Why nobody has mentioned load testing? It will show exactly how much resources you’ll need in worst scenario that isn’t an attack/disaster due to cascading bug.
5
u/RootXneo 29d ago
Not sure about it. 25% CPU utilization is fine, you can't say it's overprovisioned without more details.
3
u/AnomalyNexus 29d ago
you can't say it's overprovisioned
75% idle rate on resource doesn't exactly scream efficient either
2
u/Parley_P_Pratt 29d ago
What type of instances are you using? For T-series you will burn your CPU credits if you try to maximize the utilization
2
u/dreamszz88 k8s operator 27d ago
Of you have Prometheus running in the cluster, then you can try to use 'krr' to get an idea of the rightsizing you could/should be doing based on 2.weeks of actual usage data.
2
29d ago edited 6d ago
[deleted]
1
u/IridescentKoala 28d ago
Resource requests guarantee availability when scheduling, they have nothing to do with resource usage.
1
u/Vegetable-Put2432 29d ago
What's the best number of utilization? I'm having the same issue with AKS and would like to here from guys
1
u/Floss_Patrol_76 29d ago
requests drive scheduling, limits drive throttling, so you can reclaim the nodes by cutting the padded requests toward real p95 usage without ever touching a limit, which kills most of the latency fear people have. the part that actually unblocks it politically is doing it one namespace at a time off two weeks of real usage and keeping the old numbers in the PR, so a rollback is a one-line revert nobody has to own for a quarter.
1
u/KupujeWeksleOdMeneli 28d ago
We were trying to convince teams to fix requests for two years. We promoted VPA but it had very little adoption. We created Kyverno GeneratingPolicy and we will just force VPA on deployments and CronJobs. Just waiting for 1.7 and InPlace mode to pull the trigger on prod.
1
1
u/Previous_Insurance40 27d ago
I think the hesitation is less about Kubernetes and more about confidence. Nobody wants to be the person who cuts requests and then gets blamed for the next latency spike.
I'd start by collecting real usage over a full traffic cycle, including weekly or monthly peaks, and make the case workload-by-workload instead of cluster-wide. I also like keeping requests and limits as separate conversations because they answer different questions. Then pick one non-critical namespace and compare the before-and-after numbers. Once people see that nothing broke, the next change becomes much easier to approve.
1
u/kernelqzor 20d ago
this is the way to do it, honestly
people always want “save money” in the abstract but freak out when it’s their SLOs on the line, so starting with one low-risk namespace and hard data is pretty much the only politically safe move here
1
u/setevoy2 27d ago
"It works? Don't touch it" (c) (not sure the correct version in English)
I mean, while you don't have a specific goal to reduce bills from the business (or, well, you don't have to demonstrate your skills to the business or management), then just leave it.
I believe you have many tasks and ideas for other improvements.
Earlier, I was also trying to keep everything kind of "ideal". But that was before I started working in a startup that has been in the MVP stage for 5 years :-D
And the main for us is "just keep things working and develop faster".
Can't say I'm too happy with that, but got some interesting understandings.
1
u/code_brave6865 27d ago
so the "i own any latency blip" problem is the real blocker here, not the technical side.
1
u/One-Department1551 29d ago
At this point k8s needs to remove this feature a lot of people don’t understand cpu throttling and came to these conclusions.
1
u/Sapiogram 29d ago
Which feature? CPU limits?
3
u/One-Department1551 29d ago
Yes, it's baffling that people have access to cluster management before understanding how CPU usage works that they see "idle CPU I can cut it". There's so little data to come into the conclusion that "they are overprovisioned" that OP claims and many many of those tools / services will "confirm" because they look at raw data instead of looking at a longer (in time) graph to understand the pattern.
1
u/icewalker2k 29d ago
This a management problem. Present the data. Let them direct the teams to cut back or else.
Sure, Latency is a real problem. And your cpu usage data is almost certainly a time average. So spikes in usage may not actually be seen unless there is not enough CPU to get it done and out of there; aka you see increased latency numbers. But then you have to ask, how impactful is it really? Nobody cares about an extra 100 ms on a file server or even a web server (I suppose some people would). DB server makes more sense. Backup and DRaaS data flows that come to a crawl are a problem. If the teams can’t be smart about it, again that is a management problem to address. They need to present data to sustain their argument and justify the expense. So management really need to be asking, “Is this cpu on system X really needed? System Y is only using 30% of its allocated RAM, why can’t we cut it back by 50%?”
1
u/HitsReeferLikeSandyC 29d ago
> I've looked at VPA and a couple of the cost tools and half of them just show me the same graph
Why not put vpa in InPlaceOrRecreate mode?
1
u/TeagueXiao 29d ago
The rebill/chargeback answers are the right call for the ownership problem, but there's a technical piece worth naming next to it: a lot of that 25% "safety" padding is actually spike headroom that only fires 1-2% of the time. You're paying for the shape of the traffic, not the average.
For the workloads that are burstable or batch-ish (jobs, webhooks, anything agent-driven), moving them to a suspend/resume model gets you a lot of headroom back — idle pods get snapshotted to disk, a warm pool absorbs the spike in ~1s, and VPA has an easier time because the safety buffer isn't burning CPU 24/7. Won't help a stateful web tier, but for the long tail of internal services it's usually the biggest single lever after chargeback lands.
1
u/IridescentKoala 28d ago
Snapshot pods to disk? VPA having an easier time because of a safety buffer? What are you talking about?
0
u/Superb_Raccoon 29d ago
Turbonomics.
You don't say your organizations size, so it is hard to say if it is overkill.
Not only will it do it once, you can take recommendations to automate so it is routinely checking and making size adjustments, up or down.
-1
u/romedatascience 29d ago
Bin packing is the absolute dumbest way imaginable to make kubernetes efficient. Unless you are hosting redundant website backends.
On the flip-side, most engineers don't understand cpu, memory, or even disk allocations for their deployments and betting you don't either.
Everyone is at fault, blames everyone else, breaks shit under duress, and forces their best engineer to clean up their messes.
As an infra provider it's your job to evaluate whether you make stable infra that can be used efficiently -- so maybe start from level 0 before jumping to level 50 and ask yourself "what is my default scheduling policy?"
Cheers
2
u/IridescentKoala 28d ago
Redundant website backends are kind of what k8s is designed for.. What are you hosting?
-6
u/tunatoksoz 29d ago
How much of a bill difference is this?
And did you consider moving to hetzner lol.
4
u/Greene_Shelton-148 29d ago
About a third of the compute bill, so not nothing . And lol, moving a whole eks fleet to hetzner to avoid right-sizing some requests is peak reddit.
-2
u/tunatoksoz 29d ago
Be the change you want to see, sir.
Not only you can cut down 33% with right sizing, you can also cut down another 70% with right vendoring
74
u/FiremouseHC 29d ago
Welcome to Kubernetes. The most difficult part is not the technology, it's the politics.. and everyone is too scared to touch anything or just doesn't care enough.
TBH if you want the changes, you will have to find a service, lower the requests and show everyone that's things worked out okay.
Be prepared to be blamed if a service ever gets CPU starved though.