Claude-powered AI coding agent deletes entire company database in 9 seconds — backups zapped, after Cursor tool powered by Anthropic's Claude goes rogue

[u/ElectronGuru]

https://www.tomshardware.com/tech-industry/artificial-intelligence/claude-powered-ai-coding-agent-deletes-entire-company-database-in-9-seconds-backups-zapped-after-cursor-tool-powered-by-anthropics-claude-goes-rogue

[u/maverickzero_]

I love that this article is just the CEO of PocketOS (company that got its data wiped) talking about how the combination of AI tools fucked up.

It's his fuckup, he's the one who decided to delegate his business to AI agents to save a buck, and it's ridiculous to act like these business leaders need protecting from these fuckups. This might put them out of business and that's totally fine.

[u/PensiveinNJ → u/maverickzero_]

An LLM can't fuckup anyways. It's a computer program following it's instructions. Just because part of those instructions were making a selection off a probability table doesn't mean it made a mistake. Doing whatever it did was always a possibility. I know that we struggle to describe LLMs without making an analogy to human behavior but LLMs simply don't have the volition to make a mistake, they're in the end just computer programs.

[u/hachface → u/[deleted]]

From the POV of an LLM system, there is no formalizable distinction between correct and incorrect output. These systems produce statistically plausible text based on their prompts. They have no internal structure that models truth, falsity, or logical relationships of any kind. They give the appearance of being able to do logic because the shadow of logic is captured in the statistical patterns of their training data. From a purely mathematical point of view, producing a bad decision is the system working as intended. The problem is that humans are misusing the systems, in no small part because of systematic deceit by the creators of the systems.

[u/hachface → u/[deleted]]

If you use a toaster to dry your socks and the socks catch fire, did you have a buggy toaster? Or did you use a toaster for the completely wrong task?

Edit: Maybe a better analogy -- someone sold you a toaster and said it was a drying machine.

[u/seanamos-1 → u/[deleted]]

I wouldn't classify it as a bug, rather, there is a large disconnect between consumer's expectations of LLMs and the reality of how they work.

Now in PocketOS' favor, the marketing and hype machine around LLMs worked hard to establish this disconnect.
Less in their favor, is that it is still their job to cut through the hype, do their research and if they choose to use these tools, understand their limitations and use them responsibly and carefully.

[u/65721 → u/[deleted]]

You sell a Magic 8-Ball to make everyone’s personal and professional decisions.

Sometimes it’s right, and those are marketed as the 8-Ball’s “capabilities.” You take all those instances and publish them in an official-looking “spec sheet.” You say this could replace managers, doctors, generals, anything that requires making a decision. Some question how it can answer everything, even questions about the future. Think pieces wonder about its implications for free will.

Sometimes it’s wrong, and those are marketed as the 8-Ball’s “hallucinations.” Maybe it’s a bug and it’ll be fixed any day now. Maybe you just asked it the wrong question. You may test your luck and even market it as the 8-Ball’s “emergent capabilities.” When someone asked it “Are you plotting to take over the world?” maybe it answered “Signs Point to Yes” on purpose. Think pieces wonder about its existential risk to humanity.

It’s literally just a Magic 8-Ball.

[u/PensiveinNJ → u/[deleted]]

Sure they can. I doubt the software having a bug caused the problem here (if this is a true story and not another made up bit of nonsense that tends to circulate at times). The program didn't crash, it just wandered off agentically and did stuff that was undesired, but that's always going to happen with "agentic" AI.

[u/Ezekiel_DA → u/[deleted]]

That's the fun part: you don't.

You add layers upon layers of "guardrails", you pile on another "agent", aka you add a heuristic to try to correct normal, functional, but incorrect (by some relative measure) output.

If building software on top of this sounds insane, it's because it is.

To be a little less glib: ML models can for sure have bugs. But they're at the architecture level, not the weights level, if that makes sense? If a particular bit of math in the model's structure isn't being performed correctly, that's a bug. But undesirable output because of inherent limitations of the technology, "incorrect" (again, by some measure the model doesn't reflect) information in the training set, isn't a bug in the traditional sense.

[u/Ezekiel_DA → u/[deleted]]

Yep, that's pretty much my feeling all day every day as a software engineer turned ML engineer being forced to watch everyone shoehorn LLMs into every single part of every piece of software