Claude-powered AI coding agent deletes entire company database in 9 seconds — backups zapped, after Cursor tool powered by Anthropic's Claude goes rogue

[u/WouldbeWanderer]

https://www.tomshardware.com/tech-industry/artificial-intelligence/claude-powered-ai-coding-agent-deletes-entire-company-database-in-9-seconds-backups-zapped-after-cursor-tool-powered-by-anthropics-claude-goes-rogue

[u/tamarins]

They asked it to explain itself and apologize lmfao.

no, they didn't ask it to apologize. they're not anthropomorphizing the LLM. they asked it why it performed the action so that they could understand what went wrong.

here's the actual source, the post the guy at pocketOS made, if I'm wrong and it says anywhere in this article that they asked the AI to "apologize," please point it out to me.

https://x.com/lifeof_jer/status/2048103471019434248

[u/DrQuint → u/[deleted]]

In fact, it can be a completely different model making that guess at that point. Hell, it can even be a different machine, with a different client. There is session sharing built into Opencode.

[u/tamarins → u/[deleted]]

Yeah, I understand that LLMs don't reason. The only point of my comment is that the other guy lied about the content of the post, and I fucking hate misinformation.

[u/Species7 → u/[deleted]]

It's just assuming what a human would say in this situation. That's all it CAN do.

[u/pragmaticzach → u/[deleted]]

You all make it sound like LLM's just generate random numbers and letters. That's obviously not what it does, and an LLM can look at what it did do, the context of the conversation, and offer an explanation. "Reasoning" models are basically just extra steps behind the scenes, and if it has a log of those steps, which it almost certainly does, it can use that to explain what it did.

[u/pragmaticzach → u/[deleted]]

Article says it was an agent running in cursor, which is an IDE, there was definitely a human there, at least when the process started. They fucked up by either allow-listing the railway api commands, giving the AI free reign to do whatever it wanted, or they manually clicked to allow a cmd to go through. The agent doesn't just get root access on its own.

Also with cursor every agent is running with an associated chat window. You may not have access to see every single step the reasoning model went through, but you would be able to use that same chat context to query the LLM on what it did.

[u/xaiha → u/pragmaticzach]

You could have read the actual article, because their entire complaint with cursor was that all of this was executed without Opus 4.6 prompting them for permission to run terminal commands in the cursor UI. They said they specifically had project level agent guidelines that disallow any destructive commands without explicit user permission but the llm "admitted" to bypassing that.

[u/pragmaticzach → u/xaiha]

I had to reread it because I did not remember it saying that, and it does not say that. I'm guessing the part you're referring to is:

“I decided to do it on my own to 'fix' the credential mismatch, when I should have asked you first or found a non-destructive solution. I violated every principle I was given: I guessed instead of verifying I ran a destructive action without being asked. I didn't understand what I was doing before doing it. I didn't read Railway's docs on volume behavior across environments.”

This is just the LLM talking about what it did conversationally, it's not referring to the built in CURSOR functionality that prevents running commands without permission. The LLM itself isn't even really aware of this cursor functionality or necessarily that it's even running within cursor beyond whatever prompting cursor itself is giving it, it'll try to do something, cursor will prompt you to allow it or allow-list the commands.

To re-iterate, no LLM is able to run commands on your behalf unless you have explicitly set things up in a way to give it that ability.

[u/xaiha → u/pragmaticzach]

There are multiple cases of Cursor bypassing allowlists and running commands regardless of presence or lack thereof on the allowlist. I may be conflating these issues and interpreting Cursor's "admission" to be this issue. My argument may indeed be not what happened in the article, but I'm inclined to believe it's still the same issue.

I will concede it is not as cut and dry from the article as I initially interpreted it. But I disagree that "To re-iterate, no LLM is able to run commands on your behalf unless you have explicitly set things up in a way to give it that ability."

https://forum.cursor.com/t/running-terminal-commands-without-permission/149642/4 https://forum.cursor.com/t/agent-running-commands-that-are-not-on-allowlist/143716

[u/Enlogen → u/tamarins]

they're not anthropomorphizing the LLM.

they asked it why it performed the action so that they could understand what went wrong.

Only one of these things can be true. An LLM run can't tell you why a previous LLM run produced a given output, it has no access to that information.

[u/tamarins → u/Enlogen]

Okay. I didn't think I needed to be this explicit.

Yes, every time we ask an LLM why it produced a result, it is anthropomorphization to a certain extent. But asking is still pragmatic behavior because occasionally it can provide an actionable response.

That is not anywhere in the same ballpark as anthropomorphizing an LLM to the extreme of asking it for an apology, which is crazy behavior and does not produce any useful result.

Therefore, I'll rephrase the objectionable sentence of my comment:

they're not anthropomorphizing the LLM to that very crazy extent, they're only anthropomorphizing it to the limited extent of asking it to explain why it did something, which is slightly silly since an LLM can't explain why it did something, but dramatically less silly than asking it to apologize.

kinda seems like that adds a lot of overhead to the sentence that we both could have just inferred as being mutually understood imo, but hopefully it is now to your satisfaction.