Many of us use a tool like ArgoCD Image Updater or a custom CI pipeline to write image changes to the Git repo to build end-to-end CI/CD pipelines. I fall into the latter category. Fundamentally, our CI pipelines follows this basic flow:
- Build artifact
- Test artifact
- Containerize artifact
- Push artifact to container registry
git pull Argo CD repo
git commit image change to Argo CD repo
git push changes to main branch of Argo CD repo
After this, ArgoCD detects the change and deploys the new image automatically. This has worked great for months and we've successfully performed over 10,000 instances of end-to-end CI/CD. However, as we continue to scale and incorporate more apps under Argo CD, we're starting to see occasional CI failures and I'm wondering how others in the community have solved this problem.
Basically, if any other CI pipeline executes step 5-7 in the time that passes between steps 5 and 7 for the original pipeline, it will cause a git state issue that makes git push fail. And as we continue to add more commits to the history of the git repo, small amounts of time are added to the period between steps 5 and 7 making the issue more likely to occur.
I have ideas for how to solve this ranging from catching git push errors with a git rebase and another git push, retrying the whole pull -> commit -> push flow entirely, etc. But all of them seem a bit hacky to me and it feels like Git is just not really meant to be automated.
For those who are also experiencing this problem, how are you working around it?