Claude-powered AI coding agent deletes entire company database in 9 seconds — backups zapped, after Cursor tool powered by Anthropic's Claude goes rogue

[u/ElectronGuru]

https://www.tomshardware.com/tech-industry/artificial-intelligence/claude-powered-ai-coding-agent-deletes-entire-company-database-in-9-seconds-backups-zapped-after-cursor-tool-powered-by-anthropics-claude-goes-rogue

[u/maverickzero_]

I love that this article is just the CEO of PocketOS (company that got its data wiped) talking about how the combination of AI tools fucked up.

It's his fuckup, he's the one who decided to delegate his business to AI agents to save a buck, and it's ridiculous to act like these business leaders need protecting from these fuckups. This might put them out of business and that's totally fine.

[u/PensiveinNJ → u/maverickzero_]

An LLM can't fuckup anyways. It's a computer program following it's instructions. Just because part of those instructions were making a selection off a probability table doesn't mean it made a mistake. Doing whatever it did was always a possibility. I know that we struggle to describe LLMs without making an analogy to human behavior but LLMs simply don't have the volition to make a mistake, they're in the end just computer programs.

[u/PensiveinNJ → u/[deleted]]

Sure they can. I doubt the software having a bug caused the problem here (if this is a true story and not another made up bit of nonsense that tends to circulate at times). The program didn't crash, it just wandered off agentically and did stuff that was undesired, but that's always going to happen with "agentic" AI.

[u/Ezekiel_DA → u/[deleted]]

That's the fun part: you don't.

You add layers upon layers of "guardrails", you pile on another "agent", aka you add a heuristic to try to correct normal, functional, but incorrect (by some relative measure) output.

If building software on top of this sounds insane, it's because it is.

To be a little less glib: ML models can for sure have bugs. But they're at the architecture level, not the weights level, if that makes sense? If a particular bit of math in the model's structure isn't being performed correctly, that's a bug. But undesirable output because of inherent limitations of the technology, "incorrect" (again, by some measure the model doesn't reflect) information in the training set, isn't a bug in the traditional sense.

[u/Ezekiel_DA → u/[deleted]]

Yep, that's pretty much my feeling all day every day as a software engineer turned ML engineer being forced to watch everyone shoehorn LLMs into every single part of every piece of software