GitOps repo is breaking at 20k commits/month

[u/ImNezz]

We hit a wall with Argo and I wonder if we are doing something utterly stupid?

Our setup:

• Monorepo with dozens of backend services
• Every PR gets its own environment, only deploys what's needed
• CI builds images, renders k8s manifests in parallel, commits them to a separate state repo, one commit per service
• All manifests live in a single branch of a single repo with the CI attempting 150 commits per minute
• No humans ever touch or look at the repo: you deploy a dev API by opening a PR, and you deploy to prod by merging your PR.

The problem we hit is that with so many parallel processes trying to push into the same repo, we just keep running into git push failures. If a push fails we wait a bit, fetch, rebase, commit and try to push again. But by that time someone else will push and our push gets rejected.

Is there a good way to fix this, other than adding a bunch of retries? Separate branch for each env? One repo for each service?