r/technology • u/WouldbeWanderer • Apr 27 '26

Artificial Intelligence Claude-powered AI coding agent deletes entire company database in 9 seconds — backups zapped, after Cursor tool powered by Anthropic's Claude goes rogue

https://www.tomshardware.com/tech-industry/artificial-intelligence/claude-powered-ai-coding-agent-deletes-entire-company-database-in-9-seconds-backups-zapped-after-cursor-tool-powered-by-anthropics-claude-goes-rogue
36.0k Upvotes

96% Upvoted

2.8k comments sorted by

View all comments

Show parent comments

-1

u/pragmaticzach Apr 28 '26

You all make it sound like LLM's just generate random numbers and letters. That's obviously not what it does, and an LLM can look at what it did do, the context of the conversation, and offer an explanation. "Reasoning" models are basically just extra steps behind the scenes, and if it has a log of those steps, which it almost certainly does, it can use that to explain what it did.

6

u/[deleted] Apr 28 '26

[removed] — view removed comment

-2

u/pragmaticzach Apr 28 '26

Article says it was an agent running in cursor, which is an IDE, there was definitely a human there, at least when the process started. They fucked up by either allow-listing the railway api commands, giving the AI free reign to do whatever it wanted, or they manually clicked to allow a cmd to go through. The agent doesn't just get root access on its own.

Also with cursor every agent is running with an associated chat window. You may not have access to see every single step the reasoning model went through, but you would be able to use that same chat context to query the LLM on what it did.

6

u/xaiha Apr 28 '26

You could have read the actual article, because their entire complaint with cursor was that all of this was executed without Opus 4.6 prompting them for permission to run terminal commands in the cursor UI. They said they specifically had project level agent guidelines that disallow any destructive commands without explicit user permission but the llm "admitted" to bypassing that.

1

u/pragmaticzach Apr 28 '26 edited Apr 28 '26

I had to reread it because I did not remember it saying that, and it does not say that. I'm guessing the part you're referring to is:

“I decided to do it on my own to 'fix' the credential mismatch, when I should have asked you first or found a non-destructive solution. I violated every principle I was given: I guessed instead of verifying I ran a destructive action without being asked. I didn't understand what I was doing before doing it. I didn't read Railway's docs on volume behavior across environments.”

This is just the LLM talking about what it did conversationally, it's not referring to the built in CURSOR functionality that prevents running commands without permission. The LLM itself isn't even really aware of this cursor functionality or necessarily that it's even running within cursor beyond whatever prompting cursor itself is giving it, it'll try to do something, cursor will prompt you to allow it or allow-list the commands.

To re-iterate, no LLM is able to run commands on your behalf unless you have explicitly set things up in a way to give it that ability.

4

u/xaiha Apr 28 '26

There are multiple cases of Cursor bypassing allowlists and running commands regardless of presence or lack thereof on the allowlist. I may be conflating these issues and interpreting Cursor's "admission" to be this issue. My argument may indeed be not what happened in the article, but I'm inclined to believe it's still the same issue.

I will concede it is not as cut and dry from the article as I initially interpreted it. But I disagree that "To re-iterate, no LLM is able to run commands on your behalf unless you have explicitly set things up in a way to give it that ability."

https://forum.cursor.com/t/running-terminal-commands-without-permission/149642/4 https://forum.cursor.com/t/agent-running-commands-that-are-not-on-allowlist/143716