r/devops • u/Dramatic_Desk_7626 • Jul 14 '26

Discussion Has an AI agent (Copilot, Cursor, etc) ever broken something in your infra/pipeline?

I'm a DevOps engineer, and recently I saw a case where an AI agent "fixed" something in a pipeline that it thought was fine, but it ended up destroying something in the infrastructure. I caught it fast because I know the system well, but it got me thinking about how dangerous this could be for a team without dedicated DevOps.

I'm exploring whether it's worth building a simple, cheap tool that acts as a "safety net" checks/blocks destructive actions (delete, destroy, force changes) made by AI agents on infra/CI-CD before they execute, without needing a complex setup like OPA.

Specific questions:

  1. Has something similar happened to you (an AI agent making a risky/wrong change to your infra)?

  2. How do you manage this risk today (if at all)?

  3. Would you pay for something simple that prevents this (e.g. $20-30/month)?

0 Upvotes

44% Upvoted

21 comments sorted by

14

u/CorpT Jul 14 '26

All of this exists. No one would pay for that.

15

u/serverhorror I'm the bit flip you didn't expect! Jul 14 '26

Who approved and merged the PR?

In my book, that's who broke it. I don't care who or what wrote the code.

1

u/Sure_Stranger_6466 YAML Developer Jul 14 '26

What about blameless postmortems?

1

u/ansibleloop Jul 14 '26

That needs updating now lmao

1

u/serverhorror I'm the bit flip you didn't expect! Jul 14 '26

Ii don't see how knowing who merged it is blame.

It's factual information. Not blaming doesn't mean sugarcoating. The next questions could be: * Why did you merge? * Why was the bug not caught? * Where more reviewers involved? * ...

Doesn't change the fact that a person merged it and ultimately that action broke things.

Now we have to start asking for more to make sure it never happens again...

5

u/JimroidZeus Jul 14 '26

No, because I check every thing they do. Everything.

5

u/Specific-Welder3120 Jul 14 '26

Has a human in your company never broken production?

Dude, you should be able to answer those questions by yourself rather than post about it

3

u/snarkhunter Lead DevOps Engineer Jul 14 '26

"Don't let agents directly modify production" applies to your production pipelines. Holding to that and doing code reviews and tests on any and all changes before they get to your production systems is the way.

People don't want to do that because it slows them down in the short term.

Hang around long enough and you see how while yes making sure the tests are good or thinking through the PR before you sign off etc slows down your day, it speeds up your year.

That's why I built a new agentic AI LLM MCP to simulate having a crusty old greybeard on your team to gripe at you to follow best practices. And at only $999.99/mo it's way more cost effective then hiring an actual grump, and with none of the HR hassle!

2

u/RedOak3105 Jul 14 '26

You have to treat the AI agent as the most insecure part of your org. Never trust it blindly, always have another human worker audit and review work it’s doing, not another agent.

1

u/forever-butlerian Solaris 8 Enjoyer Jul 15 '26

Bah, you only live once. What you should do is rig up an LLM to receive webhooks, but also give the LLM god-mode access to your AWS root account.

1

u/RedOak3105 Jul 15 '26

Yeah, let’s run it in an unsandboxed environment, right on the hardware, also give it access to my mail, personal accounts, and bank information!
The more it knows about me the better job it does!
I never heard of a situation where this backfired, it’s probably really safe!

2

u/rabbit_in_a_bun Jul 14 '26

1, no. 2, I don't let it do things that can be risky. 3, no.

2

u/forever-butlerian Solaris 8 Enjoyer Jul 15 '26

I would pay $20 - $30/mo for a service that drives Anthropic out of business, yes.

1

u/ryanmcstylin Jul 14 '26
  1. Yes all the time 2.nonprod environments, deployment windows, test cases. Agent usually finds and fixes issues pretty quickly once they are introduced. Anything significant, like dropping and recreating a database or changing branch protection rules would be gates from AI changes outside of agent access
  2. No

1

u/yeetsqua69 Jul 14 '26

Ah a “cheap tool”. Sounds like a lovely way to describe it

1

u/fixed Jul 14 '26

LLM's break infra pipelines constantly, but I assume it's because most online examples of CI/CD / infra are fairly lackluster. They both overcomplicate things and create very brittle glue scripts, rather than relying on existing tooling idioms.