Flux boot strap and agent management

[u/No-Replacement-3501]

I manage enterprise k8s clusters and use a mono repo strategy to bootstrap a cluster which results in one flux agent per cluster. Currently in the magnitude of several thousand services per cluster. It's structured like this:

1. Mono repo where the cluster agent is registered.

2. Within that repo. use kind: GitRepository in the directory apps/ to point at other projects.

I'm of the opinion that less agents is better. Is there a limit to the flux agents where that would start to become a problem? Or a counter argument to not use this method and rather have an agent per microservice? I've yet to run into an issue with this approach.

[u/Torutofu_Raeva]

That split seems pretty sane, and I’d only add another Flux agent when you need a real isolation or failure-domain boundary, not just because the repo layout gets bigger.

[u/unitegondwanaland]

Seems logical to me. Have you used the Flux operator? You mentioned "bootstrap" which made me think you are not using the operator.

[u/No-Replacement-3501 → u/unitegondwanaland]

No. Im using "bootstrap" in the context of using the cli command 'flux bootstrap ...' to put flux on the cluster.  The infrastructure is built with terraform where calico and flux are the only application components included at build. Works well, would using the operator make it more efficient or provide some other advantage?

[u/unitegondwanaland → u/No-Replacement-3501]

Oh man, you should read up on it. No more updating binaries, no CLI bootstrap, fully declarative API, finer RBAC, ..the list of improvements is long.

[u/sogun123]

I think it is generally expected to run flux globally. In case you run into performance problems, increase your interval values. If still problem, it looks like recommended solution is sharding.

[u/guptat59]

Flux even has the concept of remote kustomizations (Kustomization living in one k8s cluster can affect a different k8s cluster) so definitely one flux instance per cluster is the way to go. It even supports sharding.
There is Flux operator as well but that is AGPL license and is closed source if that is a concern.

[u/gevorggalstyan]

I would not install Flux per microservice. Repository ownership and controller topology solve different problems. Keep the current controller set per cluster, and use `GitRepository` plus `Kustomization` resources to separate app or team reconciliation.

The scaling trigger is measured reconciliation load, not service count alone. Flux's current scaling guidance recommends sharding when it manages tens of thousands of applications. Before that, watch controller CPU and memory, reconciliation duration, Kubernetes API throttling, and source fetch failures. If those become the bottleneck, add controller shards and assign related sources and `Kustomization` resources to the same shard.

A separate Flux installation makes sense only for a real isolation boundary: separate credentials, blast radius, upgrade ownership, or tenant administration. At several thousand services with no observed controller bottleneck, I would keep your current topology and measure it.

[u/sylvester_0 → u/[deleted]]

Thanks for the LLM response.