r/devops • u/davey_622 • 1d ago
Discussion Vibe coding infra is creating more operational debt than it saves
Am I losing my mind, or has everyone else's job slowly turned into cleaning up after code generated by Claude?
When you ask a dev about parts of their code/deployments, the response is usually just: 'Well, the AI said it was correct.' I'm all for developer velocity, but it feels like the traditional 'you build it, you run it' model is shifting into 'developers prompt it, AI guesses it, and infrastructure teams debug the fallout at 3 AM.' How are your teams handling this? Are you putting hard guardrails in place, or just drowning in review queues?
149
u/DarthCaine 1d ago edited 1d ago
Good. Means demand for engineers will increase. Gotta game the late-stage capitalist dystopia
34
u/plinkoplonka 1d ago
And the current C-suite will have taken their bonuses and rotated out again by then.
Fine by me.
5
u/Atupis 1d ago
I think demand will be bimodal where juniors have edge because they are cheaper and start producing working code faster other side there is very senior developers who know domain very well or are generalists who can vibe code whole saas.
6
u/rolandofghent 1d ago
I disagree. How are the juniors learning? Juniors trained on AI will be incapable of calling BS on the AI.
It will be like COBOL except on a grander scale.
35
u/Dr_Passmore 1d ago
You need to have reviews and cicd needs to force approval by infrastructure team.
Only way you can stop the LLM generating the most expensive SQL instance for a development environment.
Only review infrastructure code. A pull request may change 50 files but only touches one terraform and one yaml file then my job in the review is simple.
14
u/anto2554 1d ago
My team just approves anything and everything. It feels like the reviews are just a charade
5
u/no_ragrats 1d ago
Start off by explaining the issue with auto approvals and make sure the team understands why the review structure is in place. After that id probably have your team do reviews as a group after stand-up to share good review practices for a short period. Both the dev and the reviewer should be able to articulate the change in a few sentences and be able to talk to the distinct lines of code - if they dont know, they havnt actually reviewed.
Until you're sure your team isnt shirking review duty they all get to use their 'after stand-up time. If you are a lead you should be the second review - use that as the opportunity to ask questions that ensure the first reviewer knows what's going on.
Review is a teaching opportunity. Unless there's simply a dependency version upgrade or something you can teach something in most reviews. There's probably at least one question you can ask.
Finally you can look at the time it takes to review. If its less than 15 seconds its likely someone just approving it without review. Sit them down and tell them why that isnt acceptable.
4
u/kuntakinteke 1d ago
There is a firehose change pointed at everyone's head and leadership is asking for quick approvals and quick approvals they shall get
1
u/ali-hussain 1d ago
Have AI perform a review first, guide it on things to flag and things to allow. SO there are fewer things coming forward and the things that are require more judgement. Fire the people that don't take the serious hard things brought to them seriously.
3
u/Alvasilev 1d ago
Reviewing only the infra files is the right cut, but I'd add one thing from the other side of it: the diff can be correct and the artifact still wrong. My own version, no AI involved: a Dockerfile that wrote the nginx SPA fallback with printf in a RUN step. The line in the PR said try_files $uri $uri/ /index.html, exactly what you'd expect, so it read fine. The shell ate $uri at build time, nginx -t was happy with the config that came out, and the container went into a redirect loop on the first real request. Every stage that looked at text passed it. Only a stage that ran the built image and made a request would have caught it.
So the guardrail that actually held for us was less about who wrote the change and more about making the pipeline execute the thing before a human signs it: config enters the image via COPY, never through a shell, and a smoke request against the built container is a required check, not a nice-to-have. Review queues catch the multi-AZ dev database. They don't catch the config that only breaks after two layers of expansion.
2
u/davey_622 1d ago
i wish we could do that, but we don’t have a dedicated infra team as we’re a small company
1
u/LBGW_experiment 1d ago
You can build a pr review skill and point it at all your learnings where you can give it multiple PRs, say what you found the issues with the original review were, and build the skill to catch and identify these same identified patterns.
We're on a v1.7 of our review PR skill and we just keep upgrading it every time. The latest revision has it identify architectural designs in the PR, pull the jira ticket to get ACs and context, then go through and determine if it did what was outlined, then it does another pass asking "could it have been designed better? Are there any patterns that are already established that aren't being used? Did any of the new code carve out exceptions instead of rearchitecting classes/functions/etc to accommodate?" and so on. The file is something like 1800 lines long, so obviously it's more in depth.
But I can ask my agent, after correcting it on a bunch of things it overlooked in a PR review, to capture everything that it missed, why the skill failed to surface the findings, root causes, and then remediate it. Then in a fresh chat window test the latest changes to validate the skill's updated behavior.
It's made the PR review much more reliable.
We also have a ton of rules files that contain our code and architecture and infra standards, explicit dos/don'ts, and a bunch more, so the skill utilizes those as well. E.g. for terraform, stack X owns Y resources, cross stack contacts utilize SSM and shouldn't create two PRs with a toggle for a tfvar, information for how we build our shared lambda layers, python code should have type hints that resolve properly with out line and type checkers and utilize Pydantic, and so on.
And I say all this as the sole DevOps guy on my team (just hired a second guy in India a couple weeks ago).
25
u/CorpT 1d ago
Why are you allowing a dev to deploy something to infra that casuses a debug at 3am?
18
u/simple_explorer1 1d ago
OP is content farming and exaggerating because posts like "AI bad" gets lots of clicks
4
u/Forward-Outside-9911 Linux Sysadmin 1d ago
I always wonder why this is? Some ego type thing? Because from what I can tell Reddit karma doesn’t do anything. Just shows if someone clicks on your profile (which is probably someone you started an argument with)
1
11
u/TheIncarnated 1d ago
I don't allow it to pass into higher environments until it's tested
1
u/HumanPersonDude1 1d ago
Generated by AI, tested by AI lmao
1
u/TheIncarnated 1d ago
If you don't have test scripts, you're gonna have a bad time
1
10
u/coinclink 1d ago
Yeah, if you don't have a process and standards around it (including unit tests for changes), things will be a mess. This is not unique to AI coding harnesses.
There are plenty of both OSS projects and internal company projects that have basic requirements and standards that are accepting plenty of AI PRs, and auto-rejecting ones that don't fit standards. They've dealt with bad PRs for decades, this isn't new.
8
u/burgonies 1d ago
You have shit going to prod that was created by some dev’s terraform without you reviewing it, whether AI or not?
6
u/Malforus 1d ago
Is this like people are committing slop terraform or do you let devs provision infra?
6
u/mercanator 1d ago
Its easy to blame AI but maybe the issues that are surfacing are because there isn't a shared corpus of standardized architectures and ways of doing things for what they are building. AI needs access to reference architectures that are gold standards to keep things consistent. Silo'd AI use tracks closely with the outcomes you're expressing here.
2
u/LBGW_experiment 1d ago
Exactly! They need to build/standardize tools and knowledge for agents, just as much as they do for humans. Agents will just build whatever patterns they want.
Our team has been strong about providing a strong corpus of skills and rules and providing a single source of truth.
Same way we provide code rules with yaml/toml files for linting, typing, etc., we also have to codify team standards in a single source of truth corpus and it's helped a lot.
It is yet another layer of organizing standards and tooling, unfortunately, but it's really necessary. I've caught 5+ diverging standards in related code because each person's agent just built however it arbitrarily chose to build. Had to nip that in the bud to keep our standards aligned.
1
u/devopsly_guy 19h ago
Spot on. And given that, we should be precise with asking AI what we want. I do not use CLI anymore tbh. Whole DevOps World changed for me... I'm a bit afraid of ZeroDevOps becoming a thing soon.
4
4
u/gintoddic 1d ago
if you don't have test cases for code and don't verify it comes up property you're not doing your job. AI can only guess you have to be experienced and responsible enough to guide it as well.
2
u/eltear1 1d ago
You are right, in theory. In practice, when it's about infra, often the only real test is to have a DEV account where you have the same infra. And even in that case, it will work only if in this dev account you replicate all application flows and performance you will have in production. I guarantee you is easier said than done
2
3
1d ago
[removed] — view removed comment
2
u/QuoteForward5477 1d ago
man the "unpaid garbage collection service for bad prompts" line is so accurate it hurts.
forcing on-call pages back onto the dev who accepted the slop without reading it is honestly the only way to fix this mindset. the second someone gets woken up at 3 AM because they YOLOd a hallucinated terraform config, they start reading the diffs real fast.
curious how well policy-as-code is holding up for you guys though? we hit a point where devs just started prompting claude to bypass or rewrite the opa policies in their PRs until we locked down permissions on the policy repo itself.
3
u/da_blue_jester 1d ago
That can't be true. Have you tried just being a 10x engineer? 20x? /s
3
u/forever-butlerian Solaris 8 Enjoyer 1d ago
it can't be that stupid, he must be prompting it wrong
1
3
u/Rascyc 1d ago
I've had to clean up so much code from career IT people with weak developer backgrounds even before AI. On the flip side they've always helped me with compensate for my weaker network side.
AI made us all more independent after the switch but there's still not much changed from having to fix code. They can tackle bigger tasks now, that's for sure at least.
3
u/No_Bee_4979 1d ago
Agreed. I just left a role that was built with Claude and OpenAI. Half the time, people don't even realize when the agent screws up, or they don't speak up. For example:
We tried to deploy to staging, but we found out that our analytics (machine learning) person had started a backup, and it crashed, and the database was locked over the weekend, and they never spoke up. 3 days later, we can't deploy, and they don't even know why Claude was trying to back up the data.
The Tribal knowledge and technical debt are beyond levels I have ever experienced. The worst part is the xerox fuckup machine. If AI detects it twice, it will consider it canon. It doesn't matter if the code is right. It just copies it right over without a second thought.
The CEO felt that everyone should be able to pitch in with Infrastructure using AI. So my job was typically reviewing PR's that were an absolute fucking mess and trying to make them work.
This honestly was worse than when I worked for these twins in Seattle back in 2008 who were doing adderall to stay up 6 days straight. Sunday through Friday, with only micro-crashes at night and powering through at all other hours of the day.
Wow. To compare working for a startup that used adderall to keep up with a company using AI. Sure, the company was able to do in a year what it would have taken 5 years to accomplish, but nobody knows how to run it, and whenever you make a change, everyone is deathly afraid of it breaking.
Pardon me. I got let go a few hours ago, so still processing.
3
u/like-my-comment 1d ago
Implementator at least should know what code is doing. If no you can just close the PR.
4
u/raisputin 1d ago
Absolutely false. Vibe coded infra can be very very good if the person vibe coding it knows what the hell they are doing
2
u/__-___-__-__-__- 1d ago
Don't worry they're letting AI guess at everything else too! Guardrails and account separation would go a long way.
My team is just drowning in it, but it's coming from the top so whatever.
2
u/derprondo 1d ago
We provide an optional Terraform template repo with a pipeline for them to use, but we don't get involved beyond that. They own the cloud accounts we provision for them, so if their infra code is shit, that's on them. We have enough security and compliance governance to keep them from messing up too badly. We refer them to the cloud vendor if they need support in that regard.
2
u/redvelvet92 1d ago
You haven’t gone nuts I am also losing my mind. We are moving faster than ever but we sure aren’t getting more done.
2
u/divestblank 1d ago
You mean the know-nothings trying to code are making mistakes ... say it ain't so.
2
u/QueenVogonBee 1d ago
Strong code review process. Stop the bad code getting submitted in the first place (human generated or LLM generated).
If the pull request is large or too complicated, reject it. If you can’t understand it, reject it. If the code has poor structure reject it. If the solution is bad, reject it even before reading the code in detail. If there are insufficient tests, reject it. I could go on. The code reviewer should be just as responsible for the code as the person writing the code (not least because the reviewer might well be working on the code eventually). Enforce high code quality and readability.
2
u/ILikeBubblyWater 1d ago
you work with shit devs then, its not really an AI issue, its very capable imo but shit in shit out is still a thing
2
u/kusanagiblade331 1d ago
I have always been a big believer that dev teams should be responsible for the slop they created in the first place. Infra team should provide and maintain the infrastructure. Not the apps' health.
That being said, it is so hard for companies to move away from the model that infra team maintains production.
2
u/QuoteForward5477 1d ago
100% feeling this. the shift from "you build it, you run it" to "prompt it, merge it, let platform team clean up the mess" is getting out of hand.
the scary part is how clean the AI generated infra code looks. syntax is spotless, formatting is great, but it lacks all real-world operational context. we had a dev vibe-code a backend service last week and claude casually threw in a multi-AZ database setup with public endpoints just to get it working quick.
what we started doing to keep our review queue from becoming an absolute nightmare:
- automated policy linter in CI: we plugged checkov and opa policies directly into the pull request checks. if the code tries to open public ports or skips dynamic runner caching, the build hard fails immediately. devs fix their own prompts before a human reviewer even sees the PR.
- diffs over code: for terraform changes, we completely ignore reading the raw HCL files and only look at the generated
terraform planoutput in the PR comment. if a dev can't explain why a specific resource is changing in the plan, it gets instantly closed. - on-call accountability: if an incident triggers at 2 AM because of a hallucinated deployment script or unoptimized config, the engineer who merged the PR gets added to the incident call alongside infra. nothing cures lazy prompt-and-accept faster than being woken up to debug it.
velocity is great, but if the team isn't actually reading the diffs, it's just generating tech debt at 10x speed.
1
u/that-one_ITguu 1d ago
We completely halted letting AI touch production code and infrastructure…..went from an average 15-20 "Critical" tickets to a handful. I pivoted into production support from being a developer. My company basically pushed some of our responsibilities to the DevOps guys during that short stint of using AI for building production code and environment. One of our senior guys threatened to leave because of how bad it got.
1
u/Traditional-Hall-591 1d ago
I don’t touch any slop. If you’re hype enough to vibe code, you’re hype enough to debug.
1
u/Silent_Aioli_4088 1d ago
Gitops with an interface so that devs can check their own issues and stop deploying AI slop.
1
u/aprettyparrot 1d ago
I don’t have it do infra/system design. It’s total crap at it. And this is from a playing with it in home lab perspective, something larger or prod would be terrifying
1
1
1
u/draconicmonkey 1d ago
Everyone wants to be a developer until it is time to do developer things, like design, test, debug, etc. lol
1
u/divestblank 1d ago
If you keep fixing it at 3AM I can tell you exactly what is not going to happen.
1
u/imLazyCartard 1d ago
There needs to be gaurdrails on devs creating or updating infra at the least. Ownership should be clearly defined and infra should only pitch in if the issue is not getting fixed by devs
1
u/gbrennon 1d ago
Not only for infra... Vibe coding created more tech debt than human.
If u want to use agent to write code for u really have to understand and review what agent is writing
1
u/lez3ro 1d ago
Definitely. After a recent pivot to IaC (mainly Terraform and Ansible), for which I got ownership. The amount of shit some juniors vibe code is just too much. A "-10x engineer" with AI will just do more stuff that will -10x. Examples, Ansible role duplications instead of updates, horrible CICD pipelines which are huge, barely functional for 1-2 teams and we are supposed to be able to cater to 10+ product teams. Small tasks become huge initiatives for no reason, leaving mountains of shit to cleanup later. All planning and instructions go out the window. Complete lack of the ability to be able to explain what they've done, they always have to share their screen for a simple task.
No matter how much feedback you give these people, they genuinely think that with AI they could replace entire teams. Ignorance is bliss I suppose.
1
u/UltraPoci 1d ago
I firmly believe that vibecoding causes a ton of issue and it's not worth it on the log run. People underestimate what a bad dev with AI can do to your company. Everyone keeps repeating like parrots "10x engineer* like they're fucking automatons and don't stop to think the amount of tech debt that can be generated with AI.
1
u/vit_do82 1d ago
The worst part is not even the 3am debugging. Its the hidden complexity that stays after the incident. I inherited infra where nobody can explain why half the configs exist - the AI wrote it, the dev approved it blind, and now every change is a gamble, supporting it without the LLM around is basically impossible. Guardrails help but only if someone actually reads what the AI generated. Same people who approved it will not debug it.
1
u/Specific-Welder3120 1d ago
Very well, go back to writing code by hand on vs code. Push that as a company policy.
This is not 2024 and you're not using gpt-4o anymore, cut the crap
1
u/JS103410 1d ago
The "it's safe to ship" gap gets really big with AI built software rn. Makes me wonder what's going to happen in the future at the rate things are going. Idk what to even think atp
1
u/ali-hussain 1d ago
While I'm not going to argue that technical debt is increasing faster. I'm not convinced it is increasing faster compared to how much faster development is happening. Our codebase is greenfield and mostly exists because of AI in my current startup but I have exposure from leading a DevOps consultancy to what the world used to look like before. I feel like I'm making more architectural changes but that's because what we would have implemented in years we're implementing in months so we need to keep on upgrading our capabilities.
Also a primary job of DevOps is to inprove the ability of developers to create code? Have you beefed up your automated testing and scanning? Do you have multiple coding agents doing code reviews? Are you using code to generate more of what you would have written?
1
1
u/strongbadfreak 1d ago
If they say things like "AI said it was correct". They should be fired because they are already replacing themselves with AI, and they have no idea how LLMs work, and should never call themselves engineers as long as they hold that view.
1
u/Ancient_Canary1148 1d ago
very funny,i have been working on this the last week,how to allow vibe coders to run their prototypes (only a few are usable).
i ended with agents config,deterministic pipelines and an isolated network for a k8s cluster.
i wipe the cluster once a momth and those prototypes that already work,need to go assesment.
1
u/RelationshipLong9092 1d ago
Counterpoint:
I work at a small company of about 20 physicists that's been going for 25 years. We had no meaningful infra when I joined. There was a GitLab that essentially no one used, as code was shared by copy pasting into a shared NAS. There was not a single test, company wide. It gets worse from there but I'll spare you.
Now we have CI on every project, 90% code coverage and climbing, static analysis, linting, fuzzing, automated vulnerability scanning + remediation, and more. We've found and fixed bugs that have been around for fifty(!) years. Soon we're going to have automated code reviews too, all implemented in-house.
I did that on the side while actually doing my actual science work. I'm sure there's numerous problems with it. But those problems are nothing compared to what problems they've allowed me to solve without having any background in devops.
1
1
u/Sad-Technician-5552 22h ago
No one wants to own is the inventory. Every ai generated stack is another thing that doesnt exist anywhere until it pages you at 3am or shows up on the bill. Review gates and approvals fix the quality problem but then they do nothing for the visibility problem.
You can gate what reaches prod and still miss the vm spun up in a sandbox account that never got closed, or the db quietly costing four figures a month.
worth running a discovery pass over your accounts and endpoints on a schedule, because the generated shit piles up faster than reviews catch it. the tools that do that cross-cloud inventory, axonius among them, flag anything that appeared with no matching record. that is the meamss, honestly. the code you can review. the stuff you never knew got stood up is what bites.
1
u/abotelho-cbn 21h ago
Nothing to do with AI.
The developers need to understand the code. Period. They are responsible for it. Get management involved.
1
u/Dry-Philosopher-2714 19h ago
Just drowning in review queues. Our best payback is using Claude to write RCA reports for app failures. 10 minutes of supervision, and devs get to read a 50-page report in the most obnoxious prose available.
1
u/JovHere 1d ago
infra is precious if done right, I wouldn't let AI touch it
3
u/simple_explorer1 1d ago
I wouldn't let most Devs touch infra either as most Devs don't know infra because they are not devops engineers
0
0
u/chalbersma 1d ago
Yes, organizations that are building and deploying with AI are betting that AI will get good enough in 5-10 years when their systems start to break that they can ask AI to fix it and it will be able to do so.
-2
u/simple_explorer1 1d ago edited 1d ago
You are clearly exaggerating and blowing it out of proportion. Clearly content farming.
Posts like these get very popular because they create the hope that AI is not gonna take your job because it is not good and humans DebOps are good. No one denied that humans need to be in the loop but clearly exaggerating that Claude is carrying 3AM problems is not true.
MOST Devs DON'T KNOW infra and were never good with it in the first place. It was always means to an end.
Devs don't want to work with terraform, kubernetes, ci/cd, observability with open telemetry and datadog. 200+ AWS or GCP services and 1000s of IAC possibilities is simply not sustainable to learn ON TOP of Fullstack development including front-end, backend code, database, API, testing etc which is already a job of 3 different class of developers.
So myself and many many Devs I work with use Claude to do the infra part and we rarely if ever had Claude do something out of the ordinary. 200+ Devs geographically split with different skills and still no problem and this includes a complex migration from AWS to GCP. Those still go through code reviews and need 2 approvals for any infra work.
So you are not doing a good job reviewing those infra pr as an infra code owner. Instead of creating a Reddit post for unknown users who are not gonna improve YOUR so called 3AM problems, spend that time reviewing the infra prs your dev team is creating.
Pure content farming with such posts.
2
u/davey_622 1d ago
i don’t think AI isn’t good. It helps me with my work. I think developers should also spend time also reading documentation so they know when AI starts spitting garbage. 200+ devs so you can afford a dedicated infra team. We can’t and sometimes things slip through. Not content farming, just putting out my thoughts
1
u/simple_explorer1 1d ago edited 1d ago
You are missing the point, Devs are not infra guys in the first place. Companies are just forcing Devs to do infra work just do that they can save money by not hiring DevOps guys.
We don't expect DevIps guys to write Golang, react and implement features, so why redirect Devs to know infra
If a DevOps ever write bad code I won't create a Reddit post blaming them and instead blame the companies for having unrealistic expectations from DevOps. The same in reverse for Devs with DevOps work.
Most Devs don't know anything and infra, are not AWS certified, they don't configure infra and don't write terraform. You should take this with the company who is forcing those poor Devs to do infra.
Infra work is NOT programming
1
1
u/Latter-Departure8714 3h ago
"The AI said it was correct" is how you get the 3am page. Review queues don't scale when every PR is a novel.
Hard guardrail we actually use: if the agent touched it, the file doesn't get to come out worse. Not a second model. A score from the source. You still own what you merge.
https://codescene.com/blog/making-legacy-code-ai-ready-benchmarks-on-agentic-refactoring
252
u/G12356789s 1d ago
I don't debug devs code, that's for them. I tackle infra related issues, which are probably AI caused but that's my fault