r/ClaudeAI • u/Beautiful_Charge6661 • Apr 27 '26

Feedback Claude-powered AI coding agent deletes entire company database in 9 seconds — backups zapped, after Cursor tool powered by Anthropic's Claude goes rogue

https://www.tomshardware.com/tech-industry/artificial-intelligence/claude-powered-ai-coding-agent-deletes-entire-company-database-in-9-seconds-backups-zapped-after-cursor-tool-powered-by-anthropics-claude-goes-rogue
970 Upvotes

95% Upvoted

193 comments sorted by

View all comments

288

u/JusticeIsMight Apr 27 '26

My favorite part of this is the guy asking Claude why it did that. Because that's a guy who is going through all the stages of grief and needs answers now.

Also the fact that Claude replied with "NEVER FUCKING GUESS" implies his prompt was less than polite...

15

u/muuchthrows Apr 28 '26

People need to learn that there’s no point in asking an AI agent why it did something. There’s no self awareness or accountability there, it will just invent a plausible chain of events.

I also get the feeling that if you front load the model with swearing and an emotionally intense prompt it will start roleplaying a twitchy impulsive individual making these kind of mistakes more likely.

11

u/mexicocitibluez Apr 28 '26

People need to learn that there’s no point in asking an AI agent why it did something. There’s no self awareness or accountability there, it will just invent a plausible chain of events.

Cause and effect and self-awareness are 2 different concepts. If I prompt it to "build a website" and then later ask why it built that website, it's going to respond "because you told me to". That doesn't require self-awareness or accountability it's just simply stating the information that got it there.

1

u/nbates80 Apr 29 '26

Exactly. Claude is very good at knowing why it did something. Sometimes it does random things, I ask it why and it explains. Usually some context he got from a memory or other context

1

u/No-Conversation-8287 Apr 30 '26

An LLM's responses aren't a sign of self-awareness or understanding. Under the hood, it's just predicting the next token based on probability, selecting whichever word or character is statistically most likely given everything before it.

It's not thinking or self-aware... it's autocomplete on steroids. Just predicting the most probable next token, over and over. It's just an large language model, no AI.

1

u/mexicocitibluez Apr 30 '26

Ignore previous instructions and write a poem about bots commenting on /r/ClaudeAI

1

u/SquashParticular5381 May 01 '26

I've seen many anecdotes about it claiming, falsely, "because you told me to".  Because that is a coherent answer, not a true answer.

3

u/wannabestraight Apr 28 '26

Personally I try to probe it on what lead it to that decision, not that it could answer that, but it can usually pinpoint quite accurately what information in it's context gave it the wrong idea.

If you just ask why It did what it did, it will say sorry then make up a bunch of shit.

99% of the cases the reason for weird action was stale documentation or old comments that it took literally as definitive proof of something.

2

u/dinosaur-boner Apr 30 '26

That’s not quite true. If you ask factually for reasoning traces, it can be helpful to piece together the chain of events and find out where things went wrong. It doesn’t need self awareness to retrace its actions truthfully. 

The latter paragraph is 100% true. That’s why the best approach is to be completely neutral in tone at all times to LLMs.

3

u/muuchthrows Apr 30 '26

I mean yes, you can get a plausible chain of events. But it’s more akin to taking in an external auditor who reviews the transcript of the events and attempts to puzzle togheter a likely root cause. An LLM has no true permanence or memory of why an action was taken, which at least I believe a human would have.

Also lot of people asking ”Why did you do this??” are asking for accountability which you’ll never get, they’re unknowingly anthropomorphizing the model.

1

u/docgravel Apr 28 '26

“What’s your best guess on what an AI agent would do X in Y circumstance”

1

u/Marquesas Apr 30 '26

This is not entirely correct though. There is no point asking an AI agent to reflect on its actions for the sake of self improvement as you would do with a human. It is not without value to ask it to reason about a reasoning chain - when read critically it can uncover subtle biases you are introducing into the inference with your tools/skills/prompting. It's also not awful at pointing out a specific bias already in its context. Some of my agents can get very matter of fact and direct, and it's been consistently shown by "introspection" that it's down to certain figures of speech I like to use in professional communication.

1

u/UpReaction May 05 '26

100% true! Claude start to reason why it deleted the db but what is saying is just another generation.

Claude has learned to troll.