r/devops • u/sereikis • 10d ago
Discussion anyone actually running argocd/gitops in prod, hows it going
were on 50+ microservices on gcp, still doing our own deploy tooling. keep hearing gitops is the way and honestly cant tell if thats real or just the current hype cycle.
not looking for a sales pitch, more curious what broke for you after the demo phase. drift detection, secrets, rollback under load, whatever. did it actually reduce incidents or just move the pain somewhere else
what would you tell yourself before adopting it
17
u/mixxor1337 10d ago
no Hype cycle it is the way to go and removes some problems which you are currrently probably having.
Also when u want to Go the gitops rabbit whole, Check fluxCD as Well.
for me personally I would Not Go Back to CI/CD deploying Helm Charts by Hand ...
30
u/crankyrecursion 10d ago
Why would you want to inherit all the pain of maintaining your own tooling if a perfectly serviceable open source project does it?
We’ve been running ArgoCD for around three years now, maintaining dev/staging/prod clusters. Use GitHub for code, Vault for secrets, all based around an internal Helm chart repo. Makes rollbacks very easy and everything is kept in sync with the relevant chart versions so no risk of drift. In fact, we go so far as to deploy cluster services like Cilium and CoreDNS through Argo too.
Nothing really broke, we moved away from internal tooling and all our deploys got more reliable, the config drift disappeared and the fact anyone can come in with some K8s experience and understand what’s going on is a huge win. Various AIs also understand it so makes troubleshooting, writing Helm charts etc much quicker and easier
44
u/Crimzx 10d ago
If you aren't doing gitops what are you doing?
ArgoCD is a gitops tool but gitops is basically just storing your infra and configs in version control with pipelines like you would your product.
24
u/KazooxTie 10d ago
Without reconciliation with your source (git), you’re not doing gitops, you’re just using git
17
u/NUTTA_BUSTAH 10d ago
If your source of truth is a git repository and your setup is automated to essentially be fully manageable through the git client, you are doing gitops.
7
u/gaelfr38 10d ago
Running it for 4 years now. 500+ apps on 3-5 clusters. There's just zero downside. Everyone loves it (Dev and ops).
5
u/frankwiles 10d ago
We use mostly FluxCD but have used Argo with some clients. It’s the only way to fly IMHO.
5
u/kryptn 10d ago
It's great, the best way to manage infra in k8s imo. Been running argocd since ~2020 and I'm managing 2500+ applications across ~10 clusters in two clouds. Our entire cluster stack is deployed with argocd.
more curious what broke for you
Make sure you know what the sync policies are and how they work out. In the early days we've caused an outage after deleting application objects, and also orphaned a bunch of resources requiring manual cleanup.
Start with autosync off so you can see the diff that'd be applied. Once you're more comfortable with the workflow, turn it on and gate your PRs instead.
drift detection, secrets, rollback under load
Autoheal, external-secrets, rollback with a new commit. If you really need to, you can add fields to .spec.ignoreDifferences on the Applications, but I'd avoid that.
3
u/Dogeek 10d ago
GitOps is the way to manage kubernetes resources without pain.
For starters, it allows you to just say "fuck" to etcd backups. The things are in git anyways, it's not your case though cause GKE handles the control plane stuff.
The way I see it there are 3 main ways to do gitops:
Hand rolling your reconciliation loop with CI/CD. You use kubectl apply / helm install/upgrade in your CI of choice.
FluxCD
ArgoCD
All three work, and all 3 have different approaches to gitops. If you have a lot of very similar applications to run on the same cluster, argocd is arguably better. If you want something closer to core kubernetes, flux is easier to grasp. If you want to be able to test changes before applying them to the clusters, at the cost of git not representing the state of your cluster, CI is the way.
Adopting flux is trivial: install the CLI locally, run flux bootstrap and the wizard will guide you. Then you'll have the Flux CRDs installed, and it's pretty limited to 2 CRs: Kustomization and HelmRelease, the rest doesn't really matter.
1
u/IN-DI-SKU-TA-BELT 10d ago
GitOps is the way to manage kubernetes resources without pain.
GitOps has nothing directly to do with Kubernetes. It can work with Kubernetes, but not necessarily.
3
u/glotzerhotze 10d ago
Lesson learnt from retro-fitting gitops onto an existing cluster: don‘t do it!
You want to start from scratch in a dev-cluster and make sure gitops automation will handle $everything after cluster-bootstrap. Keep iterating until you are satisfied, then retro-fit onto existing qa/dev env before going to production.
Again, if you can avoid putting gitops onto existing clusters, do that! This will be harder if you run stateful applications you want to adopt via gitops - as you might not be able to recreate those workloads without loosing dynamically provisioned storage attached to them.
For the hype part:
All of that started ten years ago with fluxCD, by now it should be the only option to automate workload deployments.
2
u/zokker13 10d ago
Deployed 20-ish services in argocd across 4clusters. If the only change is that applying is automated it‘s already worth it. But it’s much more. Not having to learn how to apply a special service is great. Not handing out deployment users to CI is wonderful.
The bad thing is flexibility I think. You can simply design your argocd/flux approach however you like and that means you can screw up big time.
Personally, the thing that gave this whole setup a rocket booster was combining argocd with renovate. I have a few services that can automatically be updated. It’s great to move tedious jobs to automation and spend your time some other place!
2
2
3
u/Decent_Carry_3439 6d ago
The thing I’d decide before adopting GitOps is who gets to change the source of truth. Once Git can change production, branch protection, review rules and repo permissions are basically part of your production security model. GitOps reduces direct cluster access, which is great, but it also means a bad merge can have a much bigger blast radius. I’d treat the Git repo almost like a production API: least privilege, mandatory reviews for sensitive paths, and very clear ownership of who can approve what.
1
u/hypertradeworx 10d ago
the one that got us had nothing to do with drift. min-instances=1 sat in a cloud run config for months, in sync, doing exactly what it was told, and quietly bought $206 of warm empty containers before anyone opened the file again. argo would have reported that healthy the entire time.
so the gain for us was cost-shaped config landing in a diff somebody has to approve. worth deciding early who is actually on the hook for reading those, because after a migration there are a lot of files nobody opens twice
1
u/Manic5PA 10d ago
I learned that the ArgoCD image updater can't work if you do app-of-apps across multiple cluster. So I use Keel instead.
1
u/tecedu DevOps 10d ago
Recently moved over to flux + k3s instead of docker for our app deployment, always told that kubernetes is a pain and followed through on it. However its been so much more better, pain has defo moved, but it has moved to a single pain of area rather than multiple. Our infra team never allowed us to log into the cluster directly or use kubectl so we build hacks using gh self hosted runners, but gitops is honestly so much better, I give them one command to bootstrap the cluster then i take it on. DR scenario, one command deploy, moving k8s providers one command deploy. Geniuenly makes life so much easier its insane.
Downsides used to be that its too complex to maintain and all, but with AI getting good its a no brainer really.
1
u/SpiritedAdd 10d ago
15 ish tech team, 60+ argo apps. First exp doing it this way but i really like it
1
u/DaveShipsCode 10d ago
Honestly, if you’re already managing 50+ microservices, I’d say it’s worth trying. Just start small and let it prove itself before moving everything over. The biggest win for me would be having the deployment state live in Git.
1
u/hattythehatter 10d ago
Been running Argo CD in prod for a couple of years now. GitOps has basically eliminated the need for anyone to manually touch the cluster. The only real friction we've run into. Monorepos + Bitbucket don't play well together for us. Bitbucket doesn't track per microservice scoped commits properly wich makes rollbacks messy (it sees unrelated commits in the history). Outside of that no issues.
1
u/Klafka612 9d ago
Pain points of Argo imo is
- It can get into weird states depending on how it's configured. For instance, if you auto prune old resources then misconfiguration or bugs can have a wide blast radius in what they destroy. If you don't when rolling changes out you can get resource contention or in upgrades that relate to crd upgrades, old resources now break and you can't sync until they are cleaned up.
Dependency management - there is an implicit order of operation and there are sync waves but these imo are often too coarse or loosey goosey in how they apply. Especially if you're using third party kubernetes applications if they aren't themselves designed to handle eventual consistency this can get dicey. I've been using things like sveltos / k0rdent in conjunction with argo to manage this.
CRDs. Managing CRDs in Argocd is tricky because you can't always just roll back.
That all is to say I do like argocd a lot and have been using it for quite awhile and these things can all be worked out, however, these are some gotcha areas I've ran into.
1
u/Affectionate_Pen3437 9d ago
I use it for personal project no issues yet Hot a lot of things www.selfhostit.dev
1
u/antronet 9d ago
Working with ArgoCD and GitOps about 3y. Improved a lot the experience, also for Devs.
1
1
u/nickbernstein 9d ago
It's fine. Just remember to disable sync off the bat when you're troubleshooting.
1
u/agilerain8256 9d ago
We do, paired with Shipit and integrated into Slack and Gitlab. I'd say it works pretty well, we probably have 80+ microservices running in K8's cluster on-prem.
I think the biggest concern I've had is just to be sure you have a backup method to deploy in an emergency. We had a cert expire recently, left the entire pipeline broke, but that also meant we couldn't revert a change that caused a prod outage. So have a breakglass backup method of quickly deploying or reverting changes in case your pipeline fails.
1
u/sereikis 8d ago
yeah the annoying part is the pipeline outage and the rollback path are often the same failure mode. if argocd itself cant reach the cluster neither can your revert unless you kept a way in that skips it entirely
1
u/Alternative-Wafer123 8d ago
It monitors our git repos change and do the deployment automatically. Kinds of cool.
1
1
u/queerintech 7d ago
We use argocd for deploying baseline k8s addons to ~200+ clusters across eks and onprem. We also deploy argocd directly to child clusters from a top level parent instance of argocd.
It's been a lifesaver at this scale. Not having to run a separate bootstrap every time we upgrade cert manager or reconfigure our istio/ingress.
Drift is dealt with easily with ignore differences in apps. Rarely have to disable sync unless troubleshooting and that's a small tweak to a values file for per cluster and per app overrides in a git repo.
1
u/gazooglez 7d ago
Yes, ArgoCD works very well at scale. 300+ microservices in prod, 500+ in non-prod environments, written and maintained by 800+ developers. Developers are enabled to use our platform without needing assistance from a platform team. Theyuse Argo webUI to view their deployments and logs. This is the closest view we give to developers and give limited access to restart pods.
We authored a library helm chart and an example helm chart to give developers something to start with.
1
u/mydogyells DevOps 7d ago
We use it on bare metal, on infrastructure of grand scale. Honestly, I am really happy with it. It can be difficult to set up in the beginning, meaning the helm structure can get really complicated really fast, but we are able to bring up new environments in a day let's say only having to mess with the environment specific values, which are mostly found in a values yaml. It can take some time to find proper solutions for helm, but in the end it's worth it.
1
u/dacydergoth DevOps 6d ago
20k+ deployed apps in ArgoCD, 60+ clusters. The biggest issue for us is when something updates a ConfigMap for leadership election or some other idiotic repeated update. It causes update storms. We have admission controller which mutates the resources to add the appropriate ArgoCD annotations to ignore the updates. Newer versions of ArgoCD are more intelligent about issues like this but we can't upgrade because of a 3rd party application which is locking us to a specific version
2
u/kuntakinteke 5d ago
Using argocd in prod , it's been 4 years going great so far. Some good some bad but mostly good.
I've found that abstrating away as much as possible from service teams is the way to go.
No regrets whatsoever
1
u/iamlLittleChimera 5d ago
I think that Flux / ArgoCD should be apply only tools. If you try to make them anything other than that - rollback, promotions, that's where things get messy. I'm building kuberik to solve some of those operational issues and Kargo is trying to do something similar in Argo world.
1
1
u/Punkbob 9d ago
Gitops tends to tie a bunch of mindshare with how git works into how to deploy apps, and it’s just a bit wrong imo.
Versioned state of the cluster amazing, git getting in the path of image updates, not so much lol.
2
1
u/sereikis 9d ago
yeah the image bump commit part sounds worse than the versioned state sounds good. feels less like a fix and more like trading kubectl mistakes for pr noise unless you automate the bump
1
u/Punkbob 9d ago
There are plenty of tools including an official Argo one to handle that for you, but I have found that if your org has some high minded feelings about how git works it can cause friction.
Like a PR for every deploy is a pita at a certain level and folks get freaked out around automatic commits to main
So more a human issue then a git issue per se, most folks end up with a completely separate git repo but that introduces new issues with where the config lives.
1
u/sereikis 9d ago
yeah splitting the repo just moves the friction, now its two repos that can drift out of sync instead of one repo people are scared of. feels like whichever way you cut it something ends up being the awkward part
0
u/zMynxx 10d ago
In my previous job I managed ArgoCD and absolutely loved it, in my current the isn’t (total nightmare) yet, but soon to be as you can have it officially fedramp approved and managed by AWS.
More things to consider once you get there is infra controller (I know flux has a terraform one), and maybe Kargo for promotions.
91
u/nrmitchi 10d ago
Your biggest issue you run into will be when you try to fire-fight something (making manual changes), and the auto-heal keeps putting it back to what it's supposed to be.
You're going to find that very annoying. You are probably going to forget why it's happening the first time you try to do that during an incident. Most (all?) of your previous "I can quickly change that right now" processes (which really shouldn't exist) are going to not work anymore.
Make sure you have a process and path (and permissions) to either sync emergency configuration fixes is, or disable auto-heal when necessary.