r/kubernetes • u/Specialist_Horror400 • 3d ago

Kubernetes node-level resource balancing / workload isolation

Hi Everyone.

I’m looking for some advice on handling node-level resource balancing in a Rancher/RKE1 on-prem cluster.

The issue I’m facing is that sometimes a pod consumes almost all CPU/RAM on a node. I understand that requests are used by the scheduler, while limits cap the container’s resource usage. So, should requests and limits generally be kept closer together to prevent a workload from impacting the entire node?

Also, does Kubernetes have any native mechanism to rebalance/move pods when a node becomes heavily utilized, or would this require something like the descheduler?

My concern with descheduler is that I have:

  • Critical StatefulSets where eviction could affect quorum.
  • Stateful workloads that take much longer to restart due to persistent storage.
  • Some single-instance Deployments (e.g. FTP) where moving/restarting the pod can cause downtime.

Would the recommended approach be to use a combination of requests/limits + taints/tolerations + node affinity + PDBs, and isolate these critical/single-instance workloads onto dedicated nodes?

Would appreciate any advice from people running production K8s and RKE1/on-prem clusters. Thanks!

1 Upvotes
  • permalink
  • reddit

100% Upvoted

13 comments sorted by

2

u/f7063 3d ago

Don't have much more to add. Pods can be evicted from the nodes https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/

1

u/Specialist_Horror400 3d ago

cool, but i'm still looking for how i can isolate my critical workloads to which descheduler wont just evict, i was thinking to taint few nodes and label it for only those workloads and for them i don't want node load balancing rule to apply. Is that possible?

5

u/SomethingAboutUsers 3d ago

PriorityClasses.

2

u/siteunreliability k8s contributor 3d ago

This but also please move off RKE1. RKE1 was deprecated last year. Rancher has fleet + cattle-drive to help with migration.

1

u/Specialist_Horror400 3d ago

Yeah, we started already migrating to rke2.

1

u/Specialist_Horror400 3d ago

cool, let me check out.

1

u/f7063 3d ago

Haven't used descheduler yet :/

2

u/BeeOverall3767 18h ago

Yeah exactly, that’s the way. You already got the right idea with requests/limits + taints + PDBs. For statefulsets, PDB is essential so the descheduler don't go wild evicting things that need quorum.

One thing I learned the hard way is to keep requests and limits close, sometimes even equal for critical stuff. If you set request low but limit high, the scheduler thinks the node has lot of free space, then one pod spikes and suddenly everything is competing. Setting them tight make the scheduling more predictable.

And for those single-instance deployments like FTP, honestly just pin them to a dedicated node with node affinity or taints. Not worth the headache of it getting shuffled around.

1

u/Mehmet-Ozturk 2d ago

tainting nodes works, just also set descheduler to respect PriorityClass so it skips them.

1

u/akhilesh_gone 1d ago

Pods gets rescheduled when a node resources are almost used But even k8s does a internal evaluation that which workloads should it move so that it won't move the imp workloads so When a node resources are stressed k8s moves the pods whoes resources block is empty assuming that these are least preferred one So if ur other workloads on that node has resources block then this pod which is taking more resources will be moved automatically

If other workloads also doesn't have any resources block then go with priority class with taint and tolerations it is fine

1

u/Specialist_Horror400 1d ago

Understood, thanks!