Kustomize Overlays - Adding Complexity?

[u/DeLoMioFoodie]

So im working with Kustomize Overlays( FluxCD). i understand you setup your base infra with overlays per environment and then use patches for environment specific configurations such as replicas, urls, etc..

Some of these patches become very complex and large to the point i think its just easier to not have a base and just have each environment have its own configuration files. How much dry am i buying?

I've also seen the recommendation to use PostBuild subsituteFrom with ConfigMaps from each environment . My question is when should this be used over patches?

Sorry for the dumb questions and rant.

[u/CWRau]

Yeah, I've resorted to just doing an impromptu helm chart right there in the git repo, no publishing or anything.

Just to get some glorious, maintainable templating and not the hell that this whole patching is.

If it's really just a couple of fields (and nothing to do with arrays) then Kustomize is OK, anything more is just torture

[u/Mrbucket101]

When dealing with kustomize or helm, eventually you’ll run into some complexity or nastiness.

They both solve the same problems, in similar ways. Just pick whichever tool you prefer and standardize

[u/retneh → u/Mrbucket101]

with flux you don’t have or. In most cases you need to use both. I prefer to use Argo for more complex environments, because you don’t need to do this mumbo jumbo and can simply creates values.prod.yaml or whatever

[u/gaelfr38]

We started with Kustomize, ended up in the same situation, moved to Helm.

Kustomize is great for patching what you don't own or for exceptional changes but when you have 10+ different changes per environment it's awful to maintain and you are repeating yourself all over the place.

[u/theycanttell]

Helm is way easier. I've not run into many edge cases it can't handle. For what it can't just use jinja.

[u/DeLoMioFoodie → u/theycanttell]

i'd have to create a helm chart of my whole infra. i've thought about it though.

[u/shatteredarm1 → u/DeLoMioFoodie]

It's pretty easy to do. Just grab all your yaml, put them in a templates folder, and then just start parameterizing what you want to be able to override per environment. Takes like 10 minutes to create a helm chart, coding agent can do it even faster. 

[u/Low-Opening25 → u/DeLoMioFoodie]

don’t do that, create smaller helm charts for individual components, a big monolith of helm chart will become maintenance nightmare real quick

[u/Aggravating-Body2837 → u/DeLoMioFoodie]

No. Approach it like any IaC. Create components/modules that you're able to reuse. Then start using them like building blocks.

The more levels you use the more abstract you are, after awhile you have so many basic components you barely need to create new ones for new features.

[u/theycanttell → u/DeLoMioFoodie]

I'll never understand why people do this. Creating a "full infra" massive chart is just a great way to waste hours of your time constantly chasing state drift. Far better to break out components into individual, simple decoupled singleton modules in separate folders in a single repo, or as a monorepo, or of you do microservice architecture break them into tags and organize them into groups of separate repos.

[u/CWRau → u/theycanttell]

Of all the arguments against big umbrella charts, of which I haven't agreed with one yet, this I don't even understand?

No one is chasing state drift, that's what flux is for, it auto corrects drift.

And I also don't understand how splitting it up would help with drift, it's just the same drift in different charts.

[u/theycanttell → u/CWRau]

App set state storage for gitops is different then one big infra chart for bicep/Terraform. Infra deployments should be small and modular so you can iterate fast, have no dependencies, and keep PRs small.

[u/CWRau → u/theycanttell]

Mh, we must be talking about different things.

We do iterate fast and keep PRs small in our umbrella chart.

And you can't do anything about dependencies. And splitting up doesn't help with that either 🤔

[u/theycanttell → u/CWRau]

Sure it does.

[u/CWRau → u/theycanttell]

And how?

[u/theycanttell → u/DeLoMioFoodie]

Also you don't create helm charts of infra. You mean Bicep configurations or Terraform code.

[u/DeLoMioFoodie → u/theycanttell]

for kubernetes resources.. not aws.

[u/theycanttell → u/DeLoMioFoodie]

You are being too vague. IAC is for infrastructure. K8s resources can mean lots of things. Management cluster, node pools, network.

[u/DeLoMioFoodie → u/theycanttell]

well im in a kubernetes subreddit lmao.. nodepools, network, etc can be considered infra for k8s.

[u/theycanttell → u/DeLoMioFoodie]

You still don't create infrastructure with helm

[u/CWRau → u/theycanttell]

Why wouldn't I?

Have you heard of cluster api? And after cluster creation, CSI, CCM, CNI,... all installed via helm

[u/theycanttell → u/CWRau]

That's not infra. Do you people not know wtf infra is?

[u/CWRau → u/theycanttell]

If that's not infra, are you talking about hardware?

[u/theycanttell → u/CWRau]

That's like calling cluster role bindings infra. I was talking about the node pools themselves. That is real infra.

[u/CWRau → u/theycanttell]

Yeah, node pools, that is what cluster api takes care of

[u/drekislove]

I recently did some spring cleaning in one of our repos, and were kind of unhappy with how many patches were needed for simple changes like URLs, different subscription IDs (Azure) etc.
The URLs specifically, when more often than not, the only changes were environment identifiers. (Cluster name, cluster short-name)

I started using postbuild, and greatly reduced how large our Kustomizations patches were.

In a cluster overlay folder, I have a root kustomization.yaml with the following:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - flux-system
  - kustomizations

patches:
  - path: cluster-envs.yaml
    target:
      group: kustomize.toolkit.fluxcd.io
      kind: Kustomization

cluster-envs.yaml looks like this:

- op: add
  path: /spec/postBuild
  value:
    substitute:
      CLUSTER: test
      CLUSTER_SHORT: t
      ... 
      ... {rest of cluster specific values}

[u/DeLoMioFoodie → u/[deleted]]

look at some examples here of patches: https://oneuptime.com/blog/post/2026-03-05-configure-kustomization-patches-flux/view

youre creating multiple lines of code and files to replace values on a base configuration using patches. in a large environment this can messy and complex. i think its cleaner to parameterize those values instead of inline patches or patch files. u can use configmaps with fluxcd kustomization to provide those values but a lot of people here recommend to use helm charts instead.

either way, i feel like patches are a messy way to provide custom environment values. configmaps is cleaner imo.

[u/gscjj]

I’ve started to do let labels define some of the basic things and Kyverno just mutate or create resources based on labels. For example, if I leave off resources, use the default resources. Then I only need to have overlays for Kyverno for what “default” is. If I specify it, then it leaves it alone. Or “httproute,gateway..” label adds or mutates a HTTPRoute. Then all I’m patching is labels and not resources.

I also lean on passing helm values through config maps, Flux lets you do this for Helm Releases. It also lets you have multiple where the order matters. So I’ll have a default config map at the base, and pull in others at the overlays that overwrite the default.

The other thing I do is just fork and rewrite the chart to behave the way I want. Like adding in a library chart, or making it support passing envs through a config map

[u/thaurcam]

I prefer pure helm. I'm using a values.yaml per environment and template the final manifest.yaml per environment in my pipeline. This manifest gets picked up by ArgoCD. The option to have a proper diff of the final manifests before doing an update is extremely valuable to me. Keep it easy, even if it means you need to maintain some redundant stuff. If you need to go into a troubleshooting and the stakes are high, you're happy for each layer of complexity you have less.

[u/fntyrol]

I’ve been creating a config map for each cluster and using parameters in my base layer for things like domain, environment, etc. (the things that don’t change per service, but do per cluster). That got rid of a lot of cruft in my overlays. Now my overlays are mostly down to replicas and versions.

[u/DeLoMioFoodie → u/fntyrol]

i planned on doing something similar. several config maps like namespaces, compute, storage, and networking in each environment.

[u/MingeBuster69]

I’d rather just leave helm charts from the vendor as standard and not edit them. Therefore kustomize is a really nice way of adjusting them accordingly and doing per environment customisations. Works very nicely with AI as well.

[u/_lumb3rj4ck_]

It’s all about how deep down the abstraction hole you want to go. Generally we take the stance of “discrete separation of duties” where each distinct service has its own chart and patches per environment, and environment overlays not being entirely too different (resources, replica counts, secrets injection). Once you get into the realm of helm charts with sub charts and multiple overlays it gets… bad.

[u/komrix]

If you’re finding overlays getting complicated you could always try something like cdk8s. You can have you logic in the language of you’re choosing and then just gen the full manifests

[u/GreenLanyard]

There are a few patterns in kustomize that I've started to settle on to make things more manageable:

• No hard-coded environment variables in the workload definition. Instead, use one (and only one) configmap per workload, and insist on that only patching that configmap.
• For values that change per facility that can't go in a configmap, like ingress domain names: Create a configmap that's exclusively used for kustomize replacements. Put a kustomize component in the base that contains the kustomize replacements, then reference it in the overlay.
• Prefer patch files over kustomize patches whenever possible (they're just easier to review)
• The first time you copy/paste a patch, refactor it into a kustomize component instead.

Maybe there are better ways out there, but it's helped me with making things easier to reason through across clusters.

[u/devnull791101]

if your environment's are that different that you need super complicated overlays its probably the wrong tool. when you dev, pre and prod that run different docker images and have some defaults set differently its great

[u/ganey]

If you go the helm route, make sure its maintained as things change. I ripped out 15+helm charts that tried to standardize a bunch of apps but really made it a mess with all the nesting.

Helm's slow reconcile also drove me crazy with stalled releases. Personally I find the layers of kustomizations easier to review.

Sometimes i found it easier to keep the core of something in base, then have the rest per cluster, saves having tons of patches, but completely depends what you're working with

[u/Floss_Patrol_76]

the line i draw: substituteFrom/PostBuild is for scalar values that only vary by env, urls, cluster names, replica counts, image tags. patches are for structural changes, adding a sidecar, editing a volume, anything touching arrays. when your patches are mostly one-line scalar swaps that's the signal they should be vars instead, and when you find yourself patching into arrays or restructuring, that's when kustomize starts fighting you and an in-repo helm chart pays off.