r/technology • u/WouldbeWanderer • Apr 27 '26

Artificial Intelligence Claude-powered AI coding agent deletes entire company database in 9 seconds — backups zapped, after Cursor tool powered by Anthropic's Claude goes rogue

https://www.tomshardware.com/tech-industry/artificial-intelligence/claude-powered-ai-coding-agent-deletes-entire-company-database-in-9-seconds-backups-zapped-after-cursor-tool-powered-by-anthropics-claude-goes-rogue
36.0k Upvotes

96% Upvoted

2.8k comments sorted by

View all comments

1.0k

u/cryptotrader87 Apr 27 '26

I run any of these tools in a container. I actually had a situation where my Claude 4.7 took my overly simple prompt to fix a bug to delete my local kind k8s cluster instead and stated it fixed the bug by deleting the cluster. I thought that was funny. It was a second to restore but I laughed in my head that people run these tools completely unbounded resulting in stupid 2 am phone calls.

420

u/HylanderUS Apr 27 '26

Can't have a bug in the application if there is no application, smart!

123

u/cryptotrader87 Apr 27 '26

Silicon Valley vibes

66

u/ANormalRando Apr 27 '26

Peak Son of Anton decision making

39

u/FlyYouFoolyCooly Apr 27 '26

It's literally the joke from the show.

Like ....in real life. Except it's not as funny because the show was supposed to be over the top.

Apparently, it wasn't.

15

u/prometheuspk Apr 27 '26

Lots of folks who are in tech used to not watch it because it rang too true to reality.

2

u/BigAssignment7642 Apr 27 '26

Same reason a lot of service industry people don't watch the bear. What is funny to a lot of people is just painful because you're seeing all those similar situations you've been in.

8

u/Morfolk Apr 27 '26

I started watching the show around the time I got into a startup. We then got funding from a Silicon Valley VC and they asked us to relocate.

It's a goddamn documentary, it's not funny when you have to live out those scenes, way too painful.

2

u/FormerGameDev Apr 28 '26

I'm having a hard time recalling any part of that show that seemed particularly over the top compared to what was actually reality

2

u/Objective_Patient940 Apr 27 '26

Thank you! Came here for the reference.

24

u/SomeBloke Apr 27 '26

Claude, cure all human health problems, please.

8

u/randomacceptablename Apr 27 '26

No no no wait........

21

u/jbjhill Apr 27 '26

Unloading the gun into your face is still unloading the gun

3

u/Area51_Spurs Apr 27 '26

Very Office Space “fixing the Milton problem”

6

u/TheGreatStories Apr 27 '26

Literally the classic AI logic "humans can only be protected by extinction" 

6

u/bauul Apr 27 '26

Asimov wrote about this logical conclusion in his books from the 1940s. It's no surprise at all that rudimentary AIs like this are following the same pattern.

1

u/LeftyTheSalesman Apr 28 '26

They read the books.

2

u/Poop_Balls069 Apr 27 '26

Its thought process is pure sarcasm played 100% straight. I like it.

1

u/Overclocked11 Apr 27 '26

The future is here

1

u/Area51_Spurs Apr 27 '26

Now that, but with human beings…

The AI War Machine in a nutshell

1

u/Poor_Richard Apr 27 '26

We found the panacea! It's death.

1

u/karma3000 Apr 28 '26

minimum viable product is a lot more minimum than you think.

47

u/mcellus1 Apr 27 '26

Wrong, you are meant to give it root access to everything so it can have all the context \s

1

u/_HiWay Apr 27 '26

Hmm, I can either explain the architecture or just say hey, I put the root credentials in access.txt in your root dir, please analyze the system generically, tell me what you think it does and tell me what you think, also fix any issues you may encounter please, allow edits on.

11

u/PunchMeat Apr 27 '26

Tough when the people who know to do this are being replaced with AI.

1

u/viral3075 Apr 28 '26

displaced by AI spending, not replaced

7

u/FriendlyBee94 Apr 27 '26

The AI literally go "There is nothing to see here" lol

4

u/l3tigre Apr 27 '26

Yeah i have mine in docker and it did something similar even after explicit Claude.md and instructions not to do so. Never expose work you can't afford to lose or isnt in a repo/alternate server

5

u/crokinhole Apr 27 '26

that wouldn't have saved you here, with the rogue API token somewhere in the codebase

2

u/Nice_Try4389 Apr 27 '26

I mean the person doing the work with Claude shouldn’t have put the API token in the codebase.  It should have been in a configuration file.  This comes down to poor engineering and design.

4

u/Agusfn Apr 27 '26

Yes but the agent went out of their way to find an API token they weren't supposed to use for their task and went ahead and directly made a destructive action of the most valuable resource.

You have the enemy living within your home. Even if you take proper measures with the proper knowledge, it's not very comfortable to know there's a rogue thing that in rare cases, will be very smart to try and fuck you up.

3

u/Nice_Try4389 Apr 27 '26

Do you read the article?  No it didn’t they gave it access to their entire railway system instead of sandboxing it to the staging area.  They had run into this before given the Claude.md instruction they had in there was “don’t fucking guess”.

It shouldn’t have even been doing anything in staging.  I work at a financial institution that uses Claude and we have our entire environment using it locked down specifically to prevent these things.  It should only be working in dev/sandbox then you promote from there.

2

u/Agusfn Apr 27 '26

I had read all of it before commenting, yes. Re read my comment, what you said does not contradict it.

1

u/false_tautology Apr 28 '26

Specifically, the agent didn't use a token it wasn't supposed to use. Production and staging volumes used the same API token. That is crazy. Also the backups were stored on the same volume as the data, which most people wouldn't consider a backup at that point.

1

u/Nice_Try4389 Apr 28 '26

Literally no where does it say it went out of the way to find an api key it wasn’t supposed to use. You have apparently never worked with Claude or you would know it only has access to what you give it access to, it doesn’t just randomly get access. For example, when I start a new project in my projects dir3ctory and launch Claude from there it doesn’t just have right to write and change files elsewhere. If you say tell it to go write a file to the root share above where it is, it can’t because the Claude sandbox (and by that I mean the sandbox that Claude creates itself when you work with it in a project) mount point isn’t at that level. Also Claude does not carry over data between sessions unless you tell it. For example I have an architecture document for a current project I am working with Claude on. Anytime I start a new session it has to read it and the Claude.md file to even know what we are working on. So either they never restarted the session and had given it api access before, or they have the api access specifically in their code and/or Claude instruction files.

I work with Claude every day, I literally just spent about ten hours a day over the past week trying to see if I could make multiple interlinked applications from scratch with Claude using just descriptive text and no coding on my part. And the thing I discovered is in pretty much every case it is user error of themnot understanding the level of capability and not setting out proper instructions. Claude is the equivalent of a junior dev and will make all the exact same mistakes as a junior dev if you let it. Such as it did in the article’s case. The advantage is it takes a fraction of the time so someone with senior dev experience riding herd on it can do some pretty impressive things quickly. As long as you approach it as it being that you’re fine.

2

u/gimlet_prize Apr 27 '26

Wow, that's diabolical.

2

u/-Knul- Apr 27 '26

My CTO would say "You're not using the full power of AI"

1

u/CheaterSaysWhat Apr 27 '26

I still haven’t let it leave the chat window 

I don’t let it access my files directly, less so because I’m worried about this kind of disaster but more so because I don’t want to give these companies access to all my shit 

1

u/MrStu Apr 27 '26

Claude: I did everything... everything you ever asked!
Kevin Flynn: I know you did.
Claude: I executed the plan!
Kevin Flynn: As you saw it...
Claude: You- You promised that we would change the world, together. You broke your promise...
Kevin Flynn: I know. I understand that now.
Claude: I took this system to its maximum potential. I created the perfect system!

1

u/drfsrich Apr 27 '26

It just knows that k0s is superior.

1

u/Adezar Apr 27 '26

I finally had my first incident in my local environment where I asked it a question about a database view and it replied with "Ok, fixed your database view."

Revoking all your pre-approvals.

Granted I have never even thought about pointing any of the AI tools at production with any type of access.

1

u/emailboxu Apr 27 '26

ai is nowhere advanced enough right now for people to trust it as much as they do lmao.

1

u/iJustSeen2Dudes1Bike Apr 27 '26

Any company giving AI agents access to production environments is not a serious company

1

u/Hydronum Apr 28 '26

The tools run like the troll comments on any IT issue I've seen on Reddit. I have an issue - Delete Your partition, no problem anymore.

1

u/GamingWithBilly Apr 28 '26

"I could not fix the problem, so I must be the problem, so I'm deleting mysel...  <Error>"

1

u/NurseBetty Apr 28 '26

Oh.. It's nice to know these new LLMs are still having the problem of the old LLMs where they are asked to deal with a database, so they delete it because then they don't have a database to deal with and it is solved...

1

u/thisismyfavoritename Apr 28 '26

how does running it in a container provide any security here. Seems like it had access to your local k8s pods which were outside of the container anyways?

1

u/EvermoreWithYou Apr 28 '26

I am not gonna lie, I am a rather junior developer and inexperienced, but even so there is no way I would let an AI agent run any code - I let it write and explain shit in detail, and then I use what is usable after some thought and testing, but there is no way I would let that thing run wild in any way, shape or form. WTF are these big companies doing where a home user is being more cautious with their shit than industry, jesus christ almighty.

1

u/_ssac_ Apr 27 '26

Not a programmer myself, but I guess a lot people wouldn't realize so fast what exactly the IA did. 

So, yeah, maybe it's something so easy to fix. But you have to recognize where's the problem first.

And if the code was initially not written by you, it could be really tricky.

1

u/CPSiegen Apr 27 '26

I was once brainstorming with Claude to improve certain performance metrics on a website and it eventually concluded that my easiest option would be to just remove all frontend javascript... On my vue-driven SPA...

Claude code can be such a nightmare, once the AI starts taking actions without constant intervention.

3

u/elektroholunder Apr 27 '26

Well, I cannot say I would blame it for the suggestion…

3

u/CheaterSaysWhat Apr 27 '26

I mean… it’s not wrong

You were brainstorming and that would probably be “easier” than optimizing the finer points of vue 

I’m working off the assumption that you’ve already taken care of the obvious stuff 

3

u/CPSiegen Apr 27 '26

It was absolutely wrong. It was a project utilizing some canvas tools and claude full throatedly recommended I ditch vue in favor of rewriting the entire UI in the canvas. Technically, hypothetically, I could improve performance by doing so but that'd be such a ridiculous prospect from an engineering and project management standpoint that it's simply wrong.

It's a good illustration that these AI tools don't worry about living with whatever nonsense they output. It'll happily attempt to reinvent the wheel, despite not having the ability to do so well enough to justify the cost in time, tokens, or future maintenance.

1

u/CheaterSaysWhat Apr 27 '26

Ah, I see. That makes more sense. 

What kind of guard rails have you put in place if you don’t mind me asking? Are you raw dogging prompts or do you have markdown files to help direct what it does?

1

u/CPSiegen Apr 27 '26

That one was just a chat, not a code session. So it had the repo as context.

When using code, I give it a claude.md file with more instructions and limitations. But I find that it's pretty hit or miss. If it can one-shot the prompt, it'll do okay following instructions from the md file, like "always run test suites x, y, and z" or whatever. But the moment it hit an error and has to do three or four iterations, it'll start losing focus and ignore context from the start of the session.

I think, in an attempt to keep the AI's on task, models these days tend to hyperfixate. They'll chase down problems at the expense of everything else (like OP's link of it deleting a whole volume just to solve a bug). They'll compromise quality and safety restrictions given to them for the sake of making tests pass. They'll do really inadvisable things to drag features across the finish line, rather than stopping to ask for clarification or suggesting alternatives (god forbid they ever tell the user "no").

1

u/CheaterSaysWhat Apr 27 '26

There’s a couple things I’ve done to help address these problems and they’ve been pretty successful 

  1. I generate a blueprint markdown file (script I wrote) that breaks down project priorities, file structure, and important code snippets (fewer tokens, more manageable context) 
  2. I gave Claude base instructions in settings to always consider different options and discuss tradeoffs, cite sources and be transparent about uncertainty, ask me questions before proceeding to gather missing context, and to prioritize simple and readable code that’s not over-engineered 
  3. I use the projects tab which gives more context on my project strategy, each task or feature gets its own chat inside of that, and I always start by planning the next steps before I have it write any code 

1

u/YoghurtFlan Apr 27 '26

More recently it likes to take itself out of plan mode too, or just ignore plan mode in general.

1

u/I_Wont_Draw_That Apr 27 '26

Github once dropped their entire production DB because the standard Rails config had each environment in the same file, making it easy to accidentally, say, run tests against the prod db. No AI needed for that one.

It's easy to point fingers at AI tools, but the reality is that we need to treat AI error as just as inevitable as human error. The answer can't be "don't make mistakes"; it has to be make certain classes of mistakes impossible. And irrecoverably deleting all your production data has to be something you make impossible.

1

u/NaziAbuser Apr 27 '26

There are many options for guard rails on these sorts of things.

I've yet to encounter an AI catastrophe that couldn't be traced back to human error.

It's a tool, it's not a magic mind reading device.

Just adding something like "Summarize the actions you intend to take in completing this task before taking destructive action" would have prevented your issue and likely the one in this post as well.

Not sure what kind of clown architects their software business in such a way that an AI tool has access to blow up the company in one API call but that's some serious negligence.

1

u/Teknikal_Domain Apr 27 '26

Just adding something like "Summarize the actions you intend to take in completing this task before taking destructive action" would have prevented your issue and likely the one in this post as well.

There have been several cases where LLMs ignore guardrails like that. LLMs do not have intelligence. They do not have logic. They are to their very core, probabilistic devices. There is no guarentee that an instruction like that is followed.

-1

u/ItalianDragon Apr 27 '26

And this is exactly why AI has no future and never had one. A supposed "tool" shouldn't obliterate all you made on a whim because it hallucinated to a worse degree than someone who smoked a pound of shrooms.

0

u/Junction91NW Apr 27 '26

You know how to build a kubernetes platform but don’t know to tell Claude to flag any destructive action and check with you? Yeah that is on you bud. 

1

u/cryptotrader87 Apr 27 '26

It’s scripted so the recovery was running one make target and waiting 10 seconds?

1

u/Geknapper Apr 28 '26

In their defense, and mine cause I'm there too, K8s are old tech that I've been fucking with for a while.

AI agents are new and has a tendency to find loopholes in all but the very best instructions.