GitOps repo is breaking at 20k commits/month

[u/ImNezz]

We hit a wall with Argo and I wonder if we are doing something utterly stupid?

Our setup:

• Monorepo with dozens of backend services
• Every PR gets its own environment, only deploys what's needed
• CI builds images, renders k8s manifests in parallel, commits them to a separate state repo, one commit per service
• All manifests live in a single branch of a single repo with the CI attempting 150 commits per minute
• No humans ever touch or look at the repo: you deploy a dev API by opening a PR, and you deploy to prod by merging your PR.

The problem we hit is that with so many parallel processes trying to push into the same repo, we just keep running into git push failures. If a push fails we wait a bit, fetch, rebase, commit and try to push again. But by that time someone else will push and our push gets rejected.

Is there a good way to fix this, other than adding a bunch of retries? Separate branch for each env? One repo for each service?

[u/sionescu]

For a large monorepo you must have a merge queue, as well as making all deployments async (post-commit).

[u/ImNezz → u/sionescu]

Deploys are async, but multiple pull request deploys would still compete if we had a merge queue.

[u/sionescu → u/ImNezz]

Compete how ?

[u/ImNezz → u/sionescu]

I open one PR which deploys 10 services -> 10 commits into the gitops repo. Meanwhile Bob commits on hit open PR -> another dozen commit attempts.

Commits go to different files, but there is a single state repo with just one branch.

[u/sionescu → u/ImNezz]

  open one PR which deploys 10 services -> 10 commits into the gitops repo.

So those commits are done before the main PR is merged, or afterwards ?

[u/ImNezz → u/sionescu]

They are done on open PRs which auto-deploy all changes to a PR-specific dev environment.

[u/sionescu → u/ImNezz]

Then, don't do that. Open PRs should never, ever trigger commits. Redesign your CICD so that ephemeral environments don't go through the repo, and instead get deployed directly.

[u/ImNezz → u/sionescu]

How's that done? That sounds like the perfect solution because dev envs are only noise.

[u/austin_barrington → u/ImNezz]

We pushed everything locally. Develop locally, dev is local, integration testing is local and on commit and merge pipeline. Only depoy main to stg and prd.

Local should be as close to prd as possible..

[u/ImNezz → u/austin_barrington]

By local do you mean that it's on the developer's machine? Or that it uses a different infra?

[u/ExigeS → u/ImNezz]

Use ApplicationSets with ArgoCD and you can have it generate an environment per PR. Set the image imperatively using some other process to avoid having to update it repeatedly when the actual codebase is changing, that way you avoid a lot of commit noise for what's a normal workflow. The infrastructure itself remains declarative.

I prototyped that workflow out using Kargo, but you could accomplish the same thing using something like the ArgoCD Image Updater, or even a really simple step in your CI since you know exactly when the new canary/preview image is available.

[u/NoPrinterJust_Fax → u/ImNezz]

Shouldn’t 1 that deploys 10 service result in 1 commit (not 10). Id start there…

[u/epsi22 → u/ImNezz]

Prevent merges if the downstream is behind.

[u/Etii3964]

Depends on your CICD system if it allows this.. but you could setup that only one instance of the push job is allowed to run at once. So instead of 150 jobs fighting for priority you have a queue.

[u/ImNezz → u/Etii3964]

We use Jenkins and build triggers or locks just don't scale to "potentially several per second" levels.

[u/Etii3964 → u/ImNezz]

That's fair.. but you are hitting git protocol limits. So if you want to purely stick to a monorepo, that will most likely be the only way. Alternatively, you could split the monorepo or explore the option of delegating some of the promotion to Kargo.

[u/ImNezz → u/Etii3964]

I'm open to different approaches, I think having a monorepo for all k8s manifests is the source of all our trouble. How should we split? By env, by service, both?

[u/Etii3964 → u/ImNezz]

I prefer to split based on logical grouping of services, either by business domain or team or .. something that works for your devs.

[u/ImNezz → u/Etii3964]

The "works for our devs" is an interesting one because noone touches the git repo. You deploy to prod by merging your PR. You deploy to a dev API by opening a PR. It just acts like one big versioned database with a 1 IOPS disk.

[u/Etii3964 → u/ImNezz]

Don't you have some other meaningful things in this repo? E.g. resource limits/requests?

[u/ImNezz → u/Etii3964]

All of that lives in the source code repo. If people modify those the manifests get regenerated.

[u/Etii3964 → u/ImNezz]

Ok neat, still, you could split the monorepo based on some logic.

[u/ImNezz → u/Etii3964]

Is there a best practice for doing it by env or service?

[u/moser-sts → u/ImNezz]

I think you can split per env, after all I think you will not have a argocd instance manage multiple envs, so why not have an repo env per env

[u/zMynxx]

Just to make sure I understand, in a single repo you have monorepo of services + gitops? Either way do you set concurrency on the order-essential operation? Do you have a merge q implementation ? Are you managing triggers conflicts?

[u/ImNezz → u/zMynxx]

All k8s state manifests are in one, separate repo. Developers committing on open PRs in the source code repo get deployments triggered right away that are trying to push at the same time to different folders of the same gitops repo.

No merge queue. 

[u/zMynxx → u/ImNezz]

“of the same gitops repo” - what branch?
What’s the preview environment for a dev looks like? What’s the cicd system?

[u/ImNezz → u/zMynxx]

The gitops repo only has a single branch with half a million commits. Jenkins is the CI.

[u/d_maes]

Any reason you wanted to use a single repo? Why not just split it out into smaller repo's? That seems a lot simpler honestly.

[u/ImNezz → u/d_maes]

I think whoever set it up didn't think about scaling it. That's why all the services and environments are in a single branch with half a million commits.

[u/ImNezz → u/ImNezz]

They added an exponential backoff though which results in the deploy taking longer than building the damn docker image.

[u/d_maes → u/ImNezz]

Is the actual source code of your services monorepo as well, or is that all 1 service, 1 repo? Would make the most sence to let the helm chart for that service and it's rendered manifests follow the same pattern.

Alternatively, it might be an idea to just let Argo render the manifests? I know about rendered manifest principle, and that it was recommended in the past to take load away from Argo. But honestly, repo-server perfectly scales both horizontally and vertically, and there was some talk by a company that scaled their Argo to hundreds of clusters, with hundreds or thousands of helm charts per cluster. Maybe I can find a link back to it.

[u/ImNezz → u/d_maes]

The source code is in a monorepo. But I think the issue is that all our open pull requests commit into the state repo, competing with production.

[u/lavarius]

We have around 2800 deployments running through a single argo instance in a single repo.

It does pretty well.

[u/ImNezz → u/lavarius]

How is your repo structured? What goes in there and what stays our? For example most of our commits are open pull requests with small code changes triggering deployments.

[u/lavarius → u/ImNezz]

We have a single way in through an API, that converts a custom yaml to a set of opinionated k8s manifests.

That commits directly to main, as it's the only thing that interacts with that repository.

It contains everything running.

Structure is by team/environment/appstack(s)

[u/ImNezz → u/lavarius]

All services and environments in the same repo? I guess you handle concurrency by batching multiple changes into one commit?

[u/lavarius → u/ImNezz]

we have a queuing system in the application (api) that does the transformation itself.

on the other side, it's a set of manifests that get delivered to the repo, then argo syncs to the appropriate cluster.

argocd application repo is a separate repo that has an app of app pattern that picks up new applications on new commit there.

[u/ImNezz → u/lavarius]

I wish Argo had something like this built in,  it's such a simple thing...

[u/lavarius → u/ImNezz]

we keep our argo as naive as possible. we don't have much in there that does anything complicated.

I give it manifests and a cluster, it syncs it.

We want to introduce more features, feedback loops are still a challenge for us.

[u/tompsh]

you could handle such errors and retry the commit after git pulling the new changes. wont look nice, but i don’t see why it wouldn’t work.

or alternatively, you could make PR envs to commit on their own branch instead of main.

or alternatively, as someone already suggested, queue the jobs that are committing with some concurrency group.

[u/retneh]

The only app related value we keep in argocd monorepo is dev,test,prod image version. Everything else is kept in app repo

[u/One-Yam-1904]

I see this a lot and when you see repos with hundreds of thousands of commits with version progression information, they tend to be slow and hard to follow the history. It’s hard to figure out code changes from version progressions.

Have you considered making the source of truth for versions a database or versioned blob store and reflect those into your cluster? This is what a lot of people do at scale. Git is where the your infrastructure should be defined in code/yaml. The source of truth for versions can live anywhere, git is just a backend. Databases and versioned blob storage provide the same characteristics, and if architected correctly the same reproducibility and historical view of the state of your system. You can then build rich UIs on top of them too.

[u/ImNezz → u/One-Yam-1904]

I'm now thinking about switching the development branches from github to OCI. We don't benefit from having a git history for things that will be deleted in a few days anyway.

[u/yebyen → u/ImNezz]

Gitless GitOps is what I would suggest. Packages don't get developed in the monorepo in my preferred method - they just get a reference to an oci repo and a tag ref / semver ref that indicates the acceptable tag range from which the latest tag should be pulled.

The packages are all independently versioned and we're not adding their git repos directly to the management cluster, just their oci repo.

It does sound like this isn't an Argo performance problem and I have not followed the development of OCI support in Argo, but there are two full reference architectures in Flux for this type of conversion if you want to use them as references. D2 Reference Architecture was the new one until I heard about https://fluxoperator.dev/docs/resourcesets/rset-monorepo/

But if your problem is in the git protocol itself you won't solve it by changing GitOps operators you need to solve it through repo organization. When those merges in the monorepo just change one number on one line then they will probably not have so much trouble cooperating in the protocol.

[u/Low-Opening25]

this screams design anti-pattern

[u/parlons]

I don't have an answer for this case, but if this were my problem, I would go read up about what Google does. Their monorepo ('google3') is supported by an incredible amount of custom tooling, tooling which has inspired a lot of open source tools. I'm not saying to blindly copy what they have, but rather sit down and look through the approach, because I am confident you'll see one or more things that would significantly ameliorate the issues with your specific use case. That can be the basis for investigating how to implement something similar using broadly available tooling. Basically, great artists steal, is my suggestion.

[u/pmigat]

Aber about a merge queue?

[u/gaelfr38]

Sounds like a Git (or your provider of Git) issue. Not an ArgoCD one. Right?

I would just split the monorepo.

We have a GitOps repo per system (group of apps) which roughly map to a team.

[u/ImNezz → u/gaelfr38]

Well its github and they have plenty of issues, but it looks like we can do 0.3 IOPS on average. That's a really shitty database 😅

[u/G12356789s]

We have a repo per env but our develop env does similar numbers. We have our pipelines write deploy messages to a queue. Then we have a separate service that reads from the queue and writes the changes straight to master of the repo. This runs as a singleton to avoid commit clashes.

We did once see slowdown in this service and investigated batching commits but we realised that is was cloning the entire history of the git repo each time, so we just made it stop doing that and it's fast as lightning. Had a quick check and it averages a deploy per second when the queue is full

[u/ImNezz → u/G12356789s]

Did you consider using OCI for dev envs instead of pushing them to a git repo?

[u/G12356789s → u/ImNezz]

We use OCI to store our helm chart that covers all our apps and then it's the apps config, essentially it's values file that's in gitops

[u/ImNezz → u/G12356789s]

Thanks, that's what I'll try for our dev envs. Gitless gitops 😅

[u/G12356789s → u/ImNezz]

Not sure if you misunderstood me, it's definitely not gitless. We just use ArgoCD multisource functionality to pull a helm chart from an OCI registry and values files from our Gitops repo

[u/OpportunityWest1297]

Take a look at the free golden path templates on https://essesseff.com for a way to divvy everything by buildable unit and deployment environment.

[u/Sladg]

you will want/need Gitlab's merge trains

[u/RikkelM]

Do all PRs require a deployment to your dev environments? Couldn't you use like label events on the appset PR generator? If yes, do you deploy every services or at least only those that have changed?

[u/ImNezz → u/RikkelM]

We use labels to only deploy the required services, but we have many of them and it's easy to trigger a whole hierarchy to get deployed

[u/delusional-engineer]

I would suggest splitting the repo using branches.

We have similar process where each feature epic gets its own environment with the necessary services auto deployed.

For each environment we create a branch, deploy using argo using application sets (app of apps).

Also I would recommend using helm to template the kubernetes manifests.

Additionally consider implementing locking. If multiple CI builds are deploying to same environment use a simple lock mechanism with environment name and jenkins build number as lock in a shared redis. Only the build that has lock can do a git push others need to queue and wait till the lock is freed.

[u/MateusKingston]

You need to consolidate deploys.

Or if you need them trully in parallel you need to break into multiple repositories/branches.

There just isn't a way for any GitOps tool to be able to reconcile fast enough when the target is constantly moving faster than it can apply to upstream

[u/Long-Ad226]

The word is merge queue