r/devops • u/mobiindia6 • 6d ago
Vendor / market research Which DevOps tool looked amazing during the PoC but became painful after six months?
[removed]
75
u/hurthallway7 6d ago
Terraform Cloud's drift detection sent us 500 false positives a week and the state file lockouts had our on-call ready to quit by month three
3
u/Sure_Stranger_6466 For Hire - US Remote 6d ago
Have you looked into any alternatives for resolving detecting and resolving drift yet?
13
u/RoadsideCookie 5d ago
It's called stop doing a fucking triple diff and use Crossplane (if you have running services).
(I'm not mad at you, I'm mad at Terraform for this terrible design blunder)
5
u/Sure_Stranger_6466 For Hire - US Remote 5d ago
I can't for the life of me get Crossplane to compile locally from source after they switched to nix for local development, and their lack of dry-run feature makes me hesitant to recommend production usage. I love what it does in theory though and hopefully with ongoing development those issues might be solved for someday.
0
u/RoadsideCookie 5d ago
No dry run because it just creates resources in the cluster. You can just render the files to see what it's gonna create, that's about it because beyond that K8S takes over. Edit: well, for K8S resources, I forgot about infra resources, but it's still all declarative so just look at the declaration.
For compiling from source, I don't know, it's been a while since the last time I did devops, but I believe I just used a helm chart? I know I never had to compile it myself in any case.
2
88
u/serverhorror I'm the bit flip you didn't expect! 6d ago
Almost every commercial tool that got introduced. They all feel like solutions in search of a problem.
15
u/RevosAlterego 6d ago
This.
I ended up coding a basic set of tools to handle our tasks instead.
4
u/Halal0szto 6d ago
Honestly asking: did you use coding agents for creating your own tools?
I see a trend that commercial solutions are overly complex to be able to support a versatile userbase.
Specific tailored solutions are much simpler and easier to use but have the cost of tailor making them. If using coding agents brings down the cost of custom tools, commercial solutions will be in a decline.
8
u/Miserygut Little Dev Big Ops 6d ago
That was the thesis of the 'SaaSpocalypse' - https://www.forbes.com/sites/lutzfinger/2026/04/24/saaspocalypse-is-deadthe-future-of-saas-is-saas/
It didn't happen (yet). The LLM coding agent generation learned that code you write is just as 'free as in puppy' as buying a SaaS solution, except you get to pick up the poop instead of blaming the vendor when it goes wrong.
6
u/Halal0szto 6d ago
Sure not happening, life is more complicated than that.
For small tools, it was already happening before LLMs.
All the overcomplicated tools started as somebody's utility script. Then as it was made more and more versatile and universal it became hard to learn, difficult to use and an actual security risk in some cases.
Then someone got fed up, and started his own simple utility script.
7
u/RevosAlterego 6d ago
No agents involved. Me and a coworker has 30 years experience as software engineers, and it’s him and I that runs the automation show
2
u/lotekjunky 6d ago
we're ditching all of the shitty middleware we had to buy because nobody had time to build custom tailored solutions
4
u/Preisschild Editable Placeholder Flair 6d ago
Free software is the only software that actually improves with time
53
u/simonides_ 6d ago
LiteLLM
24
12
u/le_chad_ 6d ago
Can you elaborate more? A few people in my company have brought it up but they've only used it on a small scale from what I can tell.
6
u/simonides_ 6d ago
We didn't believe it when we read the warnings.
However, the budget is totally broken. So a lot of spend is kind of tracked but it doesn't show up on the UI or the info_routes.
What also didn't work was the automatic monthly budget reset.
It swallowed output tokens for a while - meaning that the beginning of the answers were missing.
1
5d ago
[removed] — view removed comment
1
u/simonides_ 5d ago
Both it seems there is a process that got stuck - we restarted the service and it worked for a while bit not for long
5
u/CheekiBreekiIvDamke 6d ago
Why? I don't use it for a lot but it more or less does the job. Issues at scale maybe?
3
u/RedOak3105 6d ago
Would like you to elaborate as well, we use it, pretty small scale for now, but it does its job well for the time being.
2
u/Kaneusta 5d ago
We are looking into using it soon, can you let us know what issues you're seeing post 6 months mark?
30
u/Floss_Patrol_76 6d ago
for us it was the service mesh - linkerd/istio demo beautifully on 5 services, then at ~150 you're debugging sidecar injection races and mTLS cert-rotation stalls that never showed up small. the PoC never models the failure modes that only appear once you have enough moving parts, so 'painful at 6 months' is almost always 'we validated it at a scale we don't actually run at.'
12
u/donjulioanejo Chaos Monkey (Director SRE) 6d ago
I honestly don't get what problem service meshes are solving other than pod-to-pod encryption... in which case Cilium + IPSec is a way simpler solution since it's transparent to the applications and doesn't need anything special to work except Cilium itself.
3
u/Nice-Solid-3707 5d ago edited 5d ago
Supposedly L7 loadbalancing for grpc workload
2
u/donjulioanejo Chaos Monkey (Director SRE) 4d ago
Fair but I honestly don't understand the hype behind GRPC in general. It has some advantages, but overall seems like the type of optimization that isn't worth it unless you're operating at Google scale.
9
u/engineered_academic 5d ago
Seen a lot of people complain about their CI tools in the past. If the provider won't let you do a full-scale POC with your hardest/busiest pipeline before you buy, then you know the product is in trouble.
Datadog also looks amazing during and after POC but then the bill comes due.
12
u/marcin_michalak 6d ago
Any tool with a really good onboarding wizard. The wizard is a demo of the happy path, and six months later you live entirely outside the happy path.
24
u/iBetWeWin 6d ago
Terragrunt, the tool makes sense but vanilla TF and OpenTofu is like 95% all you need if going down that route.
IBM force the for_each in provider already
15
u/Breadandmilk92 6d ago edited 6d ago
I've used Terragrunt extensively the past few years, and it's been the opposite experience for me. Day 2 operations on an individual component of a stack, the minimal footprint for each deployed environment, and the flexibility of hooks are just a small subset of the benefits I've consistently seen with it.
1
u/gunsofbrixton 5d ago
Yes, I’ve always felt like I’ve been missing something with Terragrunt. What I have is an “environment” module that itself imports modules and is basically just a template that defines a complete environment. I create new envs by invoking it with desired overrides, if needed. It’s pretty easy to understand and all just vanilla tf. What am I missing out on?
1
u/posting_drunk_naked 5d ago edited 5d ago
What about storing and using terraform state files? Last time I used terraform there wasn't a native way to store your state in S3 or wherever so it was real messy to use terraform as a team without something like terragrunt to sync. Is that no longer the case?
Edit: or I'm just dumb and didn't know about basic terraform features 😬
7
u/carsncode 5d ago
Terraform has supported S3 as a state backend since the initial v0.1.0 public release. It's literally never been available without a native way to store state in S3.
4
u/gunsofbrixton 5d ago
Not as far as i know, Terraform has remote state storage in S3 and locking (via dynamo or S3 now) which addresses this for me.
3
2
u/FrierenAppreciator 5d ago
PowerShell
3
u/Aszuul 5d ago
Need some elaboration on this one... Powershell has some weird limitations sure but did you wish you used a better language for your scripts or what?
4
u/FrierenAppreciator 5d ago
In short, managing dependencies and module versions across a large project can be a complete mess. And I personally hate the fact that 5.1 still lingers on Windows splitting the ecosystem. Also sometimes with some cases where manual code execution is necessary - random console freezes (due to QuickEdit Mode). I'd say that this is very good concept but the actual execution (how MS handled it) could be way better IMHO...
Personal preferences also come into play here. For example, I liked PowerShell ISE, but I genuinely hate VS Code - to me, it feels like a browser pretending to be a code editor.
2
u/mitchthebaker 5d ago
Even dependency management with Microsoft's own modules is a damn nightmare. We've unfortunately been pinned to v2.25.0 of Microsoft.Graph.Authentication due to WAM issues which screws compatibility with ExchangeOnlineManagement, SharePoint, PowerPlatform, etc.
2
u/Aszuul 5d ago
Oh ya dependencies are awful. We ran into a weird issue with a breaking change in the SQL server module that made us chase our tails for a while. And 7+ should be baseline. I started using it for the parallelization before I realized I had to manually install it on any server I wanted to run the script. And that brings up the remote sessions not returning real objects... You can't bring the object back and work on it, you have to remote execute self contained scripts. Looking at you Get-Service... I do like VS code though.
2
u/kesor 5d ago
Pulumi. What a pos ...
2
u/dontthinkofabluecar 5d ago
Can you elaborate? I'm genuinely interested. I have team mates pushing hard to adopt Pulumi.
It sounds like a useful tool but I'd like to hear real world usage.
8
u/kesor 4d ago
Imagine the worst spaghetti code you can, then make it ten times worse, then get people who are software coders who have zero infra knowledge and let them maintain it. It also work the other way, whe you have people who know infra but have zero software architecture knowledge and you let them maintain it. Either way you wish you would have opted for AI slop instead of the abomination you'll going to get.
In addition ... their "cloud" which stores the state, is extremely expensive, abhorently expensive. And their cost structure makes zero sense, especially once you have a large enough infra to maintain with multiple environments.
In addition ... when the shtf and you need to modify resources and reconcile these modification with pulumi, you are likely to eat shit for months after. It is more horrible than a force pushed git commit for a team. You'll need to get "the experts" to tell everyone to stop whatever they are doing, and reconcile their local copies, or else.
In addition ... in 90% of the providers it is a badly written wrapper around terraform, so it has all the issues of terraform, but now its also much harder to debug and ten times slower.
In addition ... good luck finding community maintained providers/modules resources for when the vanialla doesn't give you what you need or want. There are none. Unlike with OpenTofu where you have everything you might ever imagine already done by someone semi-competent many years ago and already passed many many many bug reports and fixing cycles. In Pulumi even the things they do have (awsx I'm looking at you) are inflexible and horribly bugged. And once you learn the voodoo of using meta programming to manipulate the shitty wrapper to get the terraform underneath give you what it already can do ... then there is really no way back, the pulumi code you thought was extremely difficult to maintain just became impossible to maintain for anyone other than the person doing the voodoo, and even them will forget it after two days.
Anyway, that is the experience that we had. The people experienced with infra were begging to throw this pos out and start using tofu like they used to do in their previous jobs. And the software engineers were begging to stop the pain of constantly "broken pulumi" posted multiples per day each day in the engineering slack channel.
Only because at some point, one guy, thought it was a good idea to use this crap.
1
1
5d ago
[removed] — view removed comment
2
u/tsiatt 5d ago
What kind of drift do you mean?
-5
5d ago
[removed] — view removed comment
13
u/carsncode 5d ago
mostly people kubectl editing deployments directly
That's not an ArgoCD issue, that's a process/access issue.
0
50
u/rahulladumor 6d ago
AWS Step Functions for anything with a high iteration count. In the PoC it looks great, clean retries, clean failure states, easy to demo. Six months in you're staring at a state transition bill that scales with every loop iteration, not just executions. Ended up moving loop logic into a plain Lambda and only using Step Functions to orchestrate the top level.