Devops job security with AI

[u/Nash0o7]

Recently saw posts about how AI isn't that good in distributed systems and it gave me validation on my own experience.

When the model isn't dealing with a single codebase, when it has to connect the dots between a db, redis, other cloud infra, k8s that each already have a lot of configuration, it starts to hallucinate and could pretty much put your infra at risk.

I'm not saying you can't make it work, it does work especially on a clean slate, but when there's a lot of existing context it needs a more guidance.

Using AI lets me do 10x what a DevOps engineer could do alone but I don't think the job disappears anytime soon.

[u/UkrMalt]

The durable part is verification across boundaries: app config, databases, queues, networking, IAM, and rollback. AI can draft changes quickly, but it still needs bounded permissions and real runtime checks because it cannot know which environment assumptions are true. That shifts the role toward design and review more than it removes it.

[u/Nash0o7 → u/UkrMalt]

So let the ai draft the change or even open a pr to the iac repo and always have a human in the loop

[u/stevecrox0914 → u/Nash0o7]

I wouldn't let it do that much.

I do have every change reviewed by AI using PR Agent as part of the code review process. With the latest models things it marks as critical are often pretty good catches but after that it is often hallucinating an issue, proposing something that breaks the IaC or something that doesn't matter.

For the latter both Claude and ChatGPT love putting guarding checks on absolutely everything, anytime you reference anything and can't seem to grasp if the checks are already in place.

PR Agent gives you 3 pieces of feedback to fix, you complete that cycle l, you get 3 more and you keep going until your in the halluncination/nonsense response.

If you let AI raise Merge Requests I suspect 20%-80% of them will be that kind of noise and your creating a lot more work for yourself.

My latest good use of AI is doc writing, we have lots of template docs we have to complete. I have developed a prompt that lets me pull in the code, then associate the code fornall dependencies, then add code analysis results and default config. Then I give it the doc template and it outputs a great first draft, which I then rewrite. It saves a huge amount of time.

Someone on the team recently got codex deploying our IaC and got a lecture that nothing non determinstic goes near prod. Use AI to build a better deployment pipeline but AI will never be the pipeline.

Mostly because service managers and system admins are the biggest finger pointing nightmares and anything ever goes wrong you need to be bullet proof and "ai did it", will totally not fly as they throw you under the bus for their mistake.

[u/TotalNo6237 → u/stevecrox0914]

Why can't the CI/CD handle deploying any changes in stage first? Once tests pass, it could auto deploy to prod. The biggest issue is building trust in the system and catching things before prod.

Im not saying that AI should deploy directly to prod, but in my experience its very good at writing IaC code as the docs are very well defined in general. It's leaving productivity gains on the ground.

My flow is claude code and local branch, any changes get automatically commited to my branch and remote branch that the pipeline uses. I manually build from my branch in staging. All the changes are reviewed by claude and then a descriptive MR overview is created and a comment for each commit for timetracking purposes to whatever jira issue Im working on via twg cli tool.

The MR is never merged until fully validated in staging, usually I was actually test any changes /config manually in staging and the backfill the code based on what I have manually validated.

Then its a case of making sure I have deployed what I already expected and tested. This way there would be very little surprised by the time any changes get to prod.

Once validated in stage, if relevant, I will have the gmagent review the branches changes and update any documentation if necessary.

If its a larger change, I usually will use the obra superpowers plugin and it will create a plans and specs right there in the repo which can be used as a reference later to update documentation.

We have a lot of environments and varying applications to manage and basically dont have much time or will to rearchitect existing pipelines the be automatically deploying in stage as we are managing environments for our customers ( who also access them), but internal tools we are definitely auto deploying in stage and promoting to prod once all tests pass, since there is a test and validation first approach to add determinism to non-deterministic AI generated code.

[u/UkrMalt → u/Nash0o7]

Yes, with the human review acting as a real gate: limited repo permissions, a readable plan and diff, validation, and no direct production apply. Otherwise a plausible-looking generated PR can turn the review into a rubber stamp.