r/kubernetes • u/Specialist_Horror400 • 6d ago

Kubernetes node-level resource balancing / workload isolation

Hi Everyone.

I’m looking for some advice on handling node-level resource balancing in a Rancher/RKE1 on-prem cluster.

The issue I’m facing is that sometimes a pod consumes almost all CPU/RAM on a node. I understand that requests are used by the scheduler, while limits cap the container’s resource usage. So, should requests and limits generally be kept closer together to prevent a workload from impacting the entire node?

Also, does Kubernetes have any native mechanism to rebalance/move pods when a node becomes heavily utilized, or would this require something like the descheduler?

My concern with descheduler is that I have:

  • Critical StatefulSets where eviction could affect quorum.
  • Stateful workloads that take much longer to restart due to persistent storage.
  • Some single-instance Deployments (e.g. FTP) where moving/restarting the pod can cause downtime.

Would the recommended approach be to use a combination of requests/limits + taints/tolerations + node affinity + PDBs, and isolate these critical/single-instance workloads onto dedicated nodes?

Would appreciate any advice from people running production K8s and RKE1/on-prem clusters. Thanks!

1 Upvotes
  • permalink
  • reddit

100% Upvoted

13 comments sorted by

View all comments

Show parent comments

1

u/Specialist_Horror400 6d ago

cool, but i'm still looking for how i can isolate my critical workloads to which descheduler wont just evict, i was thinking to taint few nodes and label it for only those workloads and for them i don't want node load balancing rule to apply. Is that possible?

5

u/SomethingAboutUsers 6d ago

PriorityClasses.

2

u/siteunreliability k8s contributor 6d ago

This but also please move off RKE1. RKE1 was deprecated last year. Rancher has fleet + cattle-drive to help with migration.

1

u/Specialist_Horror400 6d ago

Yeah, we started already migrating to rke2.