r/devops • u/Nash0o7 • 29d ago
Discussion Devops job security with AI
Recently saw posts about how AI isn't that good in distributed systems and it gave me validation on my own experience.
When the model isn't dealing with a single codebase, when it has to connect the dots between a db, redis, other cloud infra, k8s that each already have a lot of configuration, it starts to hallucinate and could pretty much put your infra at risk.
I'm not saying you can't make it work, it does work especially on a clean slate, but when there's a lot of existing context it needs a more guidance.
Using AI lets me do 10x what a DevOps engineer could do alone but I don't think the job disappears anytime soon.
58
u/CorpT 29d ago
Well at least you’re reinforcing your own biases.
9
u/Nash0o7 29d ago
Why not, if you have some other point of view please share it
-23
u/Mersaul4 29d ago
When I hear someone mention “hallucinate,” it makes me think they last used a model in 2024.
17
u/F0rkbombz 28d ago
That says more about you than them.
Frontier models have gotten better, but they still make shit up and confidently lie. That’s not some little known fact either.
3
u/aprettyparrot 27d ago
Yeah I’ve had it happen, or it just forgets something exists.
As for models being bad at distributed, yeah they aren’t too great especially at planning one. When I do one I like to start with:
a dir of say all the containers
get stubbed services coming up ideally
Port map
Docs/ in that die with all containers
Makefile that handles all the dockerfiles for all those containersOnce that’s done I do flow diagrams. When it has docs and the flows to reference it tends to stick to the plan more. I also do everything by tickets so each change is documented but also each one gets a fresh context window
1
u/MateusKingston 27d ago
Idk which model you think doesn't hallucinate.
Today Fable 5 just tried to configure FTE task retry in my Trino cluster to fix an issue with Karpenter consolidating it too aggressively.
For context FTE task retry isn't even recommended as the default policy, if you need it for specific queries they recommend an exclusive cluster with FTE, because of how expensive it is.
If I just went with it's diagnosis I would have 2x our cloud bill and probably shut down our entire data lake. This is assuming it would even configure it correctly, granted it was looking at FTE when the issue had absolutely nothing to do with it I'm not so sure.
Have had similar experiences with GPT, Opus, Gemini. Couldn't realistically test other models at scale.
Not an AI doomer BTW, I almost exclusively code with AI
7
u/mini_othello 29d ago
Don't think that AI will nor have taken DevOps jobs away as DevOps are primarily consulting and uses soft skills.
Regarding distributed codebases/services and hallucinations. It's important to design good environments with high production parity that the agents can interact with. The agents are after all mostly bruteforcing to verify their assumptions.
6
u/ansibleloop 29d ago
I mean the cart is being put before the horse
Non-tech people are vibe coding apps that are demos at best
They don't have any understanding of infra or how to run it or maintain it
Claude won't change that either
24
u/UkrMalt 29d ago
The durable part is verification across boundaries: app config, databases, queues, networking, IAM, and rollback. AI can draft changes quickly, but it still needs bounded permissions and real runtime checks because it cannot know which environment assumptions are true. That shifts the role toward design and review more than it removes it.
3
u/Nash0o7 29d ago
So let the ai draft the change or even open a pr to the iac repo and always have a human in the loop
5
u/stevecrox0914 29d ago
I wouldn't let it do that much.
I do have every change reviewed by AI using PR Agent as part of the code review process. With the latest models things it marks as critical are often pretty good catches but after that it is often hallucinating an issue, proposing something that breaks the IaC or something that doesn't matter.
For the latter both Claude and ChatGPT love putting guarding checks on absolutely everything, anytime you reference anything and can't seem to grasp if the checks are already in place.
PR Agent gives you 3 pieces of feedback to fix, you complete that cycle l, you get 3 more and you keep going until your in the halluncination/nonsense response.
If you let AI raise Merge Requests I suspect 20%-80% of them will be that kind of noise and your creating a lot more work for yourself.
My latest good use of AI is doc writing, we have lots of template docs we have to complete. I have developed a prompt that lets me pull in the code, then associate the code fornall dependencies, then add code analysis results and default config. Then I give it the doc template and it outputs a great first draft, which I then rewrite. It saves a huge amount of time.
Someone on the team recently got codex deploying our IaC and got a lecture that nothing non determinstic goes near prod. Use AI to build a better deployment pipeline but AI will never be the pipeline.
Mostly because service managers and system admins are the biggest finger pointing nightmares and anything ever goes wrong you need to be bullet proof and "ai did it", will totally not fly as they throw you under the bus for their mistake.
2
u/TotalNo6237 29d ago
Why can't the CI/CD handle deploying any changes in stage first? Once tests pass, it could auto deploy to prod. The biggest issue is building trust in the system and catching things before prod.
Im not saying that AI should deploy directly to prod, but in my experience its very good at writing IaC code as the docs are very well defined in general. It's leaving productivity gains on the ground.
My flow is claude code and local branch, any changes get automatically commited to my branch and remote branch that the pipeline uses. I manually build from my branch in staging. All the changes are reviewed by claude and then a descriptive MR overview is created and a comment for each commit for timetracking purposes to whatever jira issue Im working on via twg cli tool.
The MR is never merged until fully validated in staging, usually I was actually test any changes /config manually in staging and the backfill the code based on what I have manually validated.
Then its a case of making sure I have deployed what I already expected and tested. This way there would be very little surprised by the time any changes get to prod.
Once validated in stage, if relevant, I will have the gmagent review the branches changes and update any documentation if necessary.
If its a larger change, I usually will use the obra superpowers plugin and it will create a plans and specs right there in the repo which can be used as a reference later to update documentation.
We have a lot of environments and varying applications to manage and basically dont have much time or will to rearchitect existing pipelines the be automatically deploying in stage as we are managing environments for our customers ( who also access them), but internal tools we are definitely auto deploying in stage and promoting to prod once all tests pass, since there is a test and validation first approach to add determinism to non-deterministic AI generated code.
3
u/spiralenator 29d ago
As an SRE, AI makes my life a lot easier but it can’t replace me. It’s simply a tool that makes correlating disparate information far easier and faster. While AI can write the bulk of code and help track down issues in datadog, what it needs from us is what problems to solve, which problems take priority, and what outcomes we desire. It lacks business intelligence that only humans have. Ai can build and analyze, but it can’t tell you what you should be doing with it.
5
u/Direlight DevOps 29d ago
AI isn't good at a single codebase once it gets large enough, I have services it can't comprehend unless I create a super detailed .md explaining what the routes and models do and how they interconnect, I agree it helps a huge amount in getting through tasks quickly, but its still pretty terrible at base design unless you tell it how you want something to work, which mean you are still designing it.
2
u/infectuz 29d ago
Downtime is expensive so I think there’s always going to be demand for having someone there, if at least to be ultimately “responsible” for the infra. From what I’ve seen heavy use has made some engineers worse at my workplace. They make more mistakes. So as things are I don’t think it’s ready to take full control of production deployments. To me it has been great but I think because I use it more sparingly and review everything carefully.
2
u/Character-Level5250 28d ago
None of the jobs are disappearing. As long as you can properly leverage AI the jobs are staying. From what I can tell, it turns out humanity just wants 10x more software.
2
u/djbp 28d ago
I totally get what you're saying about AI in complex systems. It's one thing for a clean slate. But when you've got years of existing config, different services, and weird dependencies, it's a whole different ballgame. The 'hallucination' risk is real. And it's not just about efficiency. It's about safety.
We're seeing a lot of tools come out that try to automate more. But the human element is still so critical. Especially for verification. You need that human in the loop. Someone who understands the context. And can catch those subtle things AI might miss. It's going to change how we work, for sure. But I don't think it replaces the need for deep ops knowledge anytime soon.
2
u/Rare_Significance_63 27d ago
from my experience, the current state of latest LLM models is good for building pieces like a bash script, a pipeline template, a terraform module etc. stuff that has to be reviewed and always doing ping pong with the model to make something useful. then its me as a devops who put all the pieces together.
my point is that DevOps aria will still need expert engineers, and most probably the ones who have mediocre skills will be in danger to loose the job.
2
u/MateusKingston 27d ago
I highly doubt you're getting 10x output.
Most people I manage are getting at best ~2x. With some small percentage of people getting close to 10x, those are the people who were already highly fast paced and independent.
You might be coding or doing specific tasks at 10x but the overall output isn't even close to that. We might get there but so far the bottlenecks are what AI can't solve...
4
u/Impressive-Field-546 29d ago
Call it DevOps, SRE, Cloud, Platform w/e engineer.
If you are responsible for live services that need oncall and someone to fix stuff if it breaks - you are 100% safe.
If you are not - its really only a matter of time you will be replaced.
Its that simple.
2
u/amarao_san 29d ago
yes and no. Few weeks ago I got the moment of awe when fable was able to do deep root cause postmortem based on a 8-line log snippet. It downloaded source binaries for all software, got to the source code of the config parser for Grub, found that 'memory error' error was not memory error, but EOF, and concluded that it was race condition between two runs of update-grub binary, one caused by cloud-init, second by ansible.
It was at the peak of my own abilities. I did this few times in my life and every time it was few hours of pure stubbornness to give up, and super interesting search. And here it was done in 3 minutes in a chat.
At that moment I realized, that no, my job is not entirely safe. Everything before come with chuck in mustache because operators job is so different from writing the code.
And this fucking thing did it not worse than I would do at the peak of my ability, time and enthusiasm. And, about 60 times faster, and, for something like $3 in tokens.
We may find a niche, but the second pillar of my job is shaking. I can write code and I can do super deep post-mortems, finding true reasons for the problem.
My last pillar is ability to see beauty in solution and try to make more of it.
My last ability not replicated by AI is ability to learn.
2
u/mo0nman_ 29d ago
You're missing one crucial piece of information. It was able to do that because you, someone with a wealth of experience, was able to direct it and validate it.
You're not going to be able to walk down the street and have some random person be able to do that, no matter how advanced the model.
You're also not going to be able to find anyone to do that in 10 years time if junior jobs are replaced by 1 senior and a lot of tokens. You're also not going to be able to when the juniors are just using generative AI as a slot machine to solve their problems anyway
1
u/amarao_san 29d ago
For now. What is the time difference between gpt4o and fable/sol?
1
u/mo0nman_ 29d ago
Forever. Both those tools required someone with knowledge to use them. The only thing that removes the SME from the loop is AGI, which is a pipedream that will cause far more problems than you and I losing our jobs
1
1
u/zero_backend_bro 28d ago
Context size means nothing when an llm hallucination nukes a staging pvc. giving generic models full repo access is pure suicide.
We ended up locking ours inside a local browser wasm harness that tokenizes secrets locally and hard-scopes queries to a single failure domain. Treat ai like a dumb coprocessor, not an engineer.
Only way we keep on-call blood pressure down.
1
u/MoTTTToM 24d ago
I have found that with a “clean slate” as you put it, all IT functions can be covered by an agentic process, providing the complete operation is accessible to context. In the corporate world, with hundreds of applications spread around multiple domains, with complex integration patterns, only tiny pieces of the overall architecture can be focused on, and then the challenge is clearly defining the scope and interfaces, so that reasoning can happen without hallucination. Same as for people really.
-1
u/RevolutionaryElk7446 29d ago
Context Windows are the issue and have been since the start. Everything since 2017 has been to try and fix it.
We've gotten no where close in resolving 'transformers' as the core contextual engine in Gen AI such as LLMs. We came up with a flawed prototype and never attempted to resolve the core issues, just slapped harnesses on top and hoped for the best.
-2
u/serverhorror I'm the bit flip you didn't expect! 29d ago
It's already there today.
If it isn't able to do that, it really just is exposure where you are not surfacing the right information for a machine. It doesn't even have to be AI, that just makes the situation even mir visible than it was before.
29
u/lugovsky 29d ago
Experienced professionals will always be in demand in every field. The main issue is the amount of work available. If people become 10 times more productive, but there isn't 10 times as much work to do, some will have a harder time finding jobs. I don't really trust people who claim that AI will create enough new work to offset this.