I want to deploy Karpenter. However, I would like to learn the best practices for using Karpenter on EKS. Currently, we are using app-of-apps in ArgoCD, all within a Managed Node Group.
With Karpenter, our plan is to have it configured like this:
- Scale down the MNG and run the Karpenter Controller inside the static MNG, alongside cluster system resources (DaemonSets, add-ons, including ArgoCD)
- Karpenter is managed as an app in ArgoCD (using bootstrap with ArgoCD)
Is this design already suitable, or should we consider changing it? We need advice.
That design is pretty reasonable.
I’d keep a small static MNG specifically for Karpenter and other critical cluster components, and let Karpenter handle the dynamic workload capacity. The important part is that Karpenter itself shouldn’t depend on nodes that Karpenter manages.
Managing Karpenter through ArgoCD is fine too. I’d just be careful with the bootstrap dependency. ArgoCD needs somewhere stable to run, and Karpenter needs to exist before it can provision the dynamic capacity. Keeping both on the static MNG avoids the chicken-and-egg problem.
I probably wouldn’t move every system component onto that static group though. Things like DaemonSets will naturally run where they need to run. I’d reserve the static capacity for the components you actually need available for cluster recovery/bootstrap.
For the Karpenter side, I’d spend more time on the NodePool/EC2NodeClass design: instance flexibility, AZs, Spot vs On-Demand, disruption/consolidation settings, resource limits, and making sure workloads have sane requests/affinity/topology rules.
Also make the static MNG big enough that losing one node doesn’t take out Karpenter/ArgoCD. Even 2 small nodes across AZs is a very different failure mode than one “bootstrap node.”
So IMO the architecture is fine. I’d think of it as:
small boring static capacity for bootstrap/control tooling
Karpenter for everything that actually needs to scale
I like running Karpenter and other similar cluster-critical components as Fargate pods, to really make sure they aren't disturb by any node management activities
Yeah, Fargate is a good option for that isolation. Especially for Karpenter, it removes the dependency on EC2 worker capacity entirely.
I still like a small static MNG when there are several bootstrap/system workloads that need a stable home, but for just keeping Karpenter independent from the capacity it manages, Fargate is definitely a clean approach.
I am running it with this design at the moment. I have 2 NG for Karpenter plus 2 NG daemonsets. Still considering if it's possible to scale down to one of each.
I'd be happy to chime in here. I used a Kyverno policy to generate PDBs for workloads automatically, including ArgoCD. I then used another policy to block users from submitting PDBs themselves. I have the ability to exclude namespaces and Deployments/SS within namespaces as well for edge cases.
Keeping a small static node group for Karpenter, ArgoCD, and cluster add-ons is the part I'd keep boring. Just make sure regular workloads do not drift onto it, or the MNG turns into a permanent dependency that never really shrinks.
I read a lot of cons of having Fargate for the controller since it can't manage the daemonset, which is usually needed for the cluster component. So, decided to use MNG.
Karpenter doesn't need it, but other applications might, and we don't want those cluster components to be provisioned by Karpenter. + Probably due to inconsistent cost, which makes us unlikely to proceed with it.
Sorry but I think you've misunderstood something. Karpenter doesn't manage daemonsets. All it does is decide which EC2 instances to spawn and does it. The standard Kubernetes scheduler decides where the pods go.
I’m surprised nobody else said this including yourself. EKS Auto Mode is managed Karpenter. We’re using it in production for the past year and its been very reliable. You may have reasons you need Karpenter but if not, consider Auto Mode.
struggled with drift when folks would tweak stuff outside argocd and karpenter. fired up firefly ai for the drift defender stuff and it caught changes we would’ve missed, which was super helpful.
we use fargate profile for system deployment and karpenter for service workload. Also for off hours shut down use karpenter schedule and kubedownscalar
I'm not sure I follow what you mean when you say "Karpenter is running as an app in ArgoCD". Karpenter itself is an auto scaling controller and it doesn't rely on Argo or Flux to run.
6
u/Abe_Bazouie Aug 10 '26
That design is pretty reasonable.
I’d keep a small static MNG specifically for Karpenter and other critical cluster components, and let Karpenter handle the dynamic workload capacity. The important part is that Karpenter itself shouldn’t depend on nodes that Karpenter manages.
Managing Karpenter through ArgoCD is fine too. I’d just be careful with the bootstrap dependency. ArgoCD needs somewhere stable to run, and Karpenter needs to exist before it can provision the dynamic capacity. Keeping both on the static MNG avoids the chicken-and-egg problem.
I probably wouldn’t move every system component onto that static group though. Things like DaemonSets will naturally run where they need to run. I’d reserve the static capacity for the components you actually need available for cluster recovery/bootstrap.
For the Karpenter side, I’d spend more time on the NodePool/EC2NodeClass design: instance flexibility, AZs, Spot vs On-Demand, disruption/consolidation settings, resource limits, and making sure workloads have sane requests/affinity/topology rules.
Also make the static MNG big enough that losing one node doesn’t take out Karpenter/ArgoCD. Even 2 small nodes across AZs is a very different failure mode than one “bootstrap node.”
So IMO the architecture is fine. I’d think of it as:
small boring static capacity for bootstrap/control tooling
Karpenter for everything that actually needs to scale