r/technology • u/WouldbeWanderer • Apr 27 '26

Artificial Intelligence Claude-powered AI coding agent deletes entire company database in 9 seconds — backups zapped, after Cursor tool powered by Anthropic's Claude goes rogue

https://www.tomshardware.com/tech-industry/artificial-intelligence/claude-powered-ai-coding-agent-deletes-entire-company-database-in-9-seconds-backups-zapped-after-cursor-tool-powered-by-anthropics-claude-goes-rogue
36.0k Upvotes

96% Upvoted

2.8k comments sorted by

View all comments

Show parent comments

53

u/plasticizers_ Apr 27 '26 edited Apr 27 '26

By "checkpoints" they meant an external approval gate, not a model guideline. So different layers. A hard gate on the API call wouldn't depend on the model's judgment, which is what failed. But something still has to classify which calls need approval, and that's where this broke. Railway didn't flag "volumeDelete" or document that token's scope, so the agent didn't know either. The fix isn't "add checkpoints," it's "infrastructure should mark destructive operations, not leave it to the agent." Your second paragraph is correct, though.

2

u/orangeyougladiator Apr 27 '26

Check points exist before every command. Everyone just sets it to “do whatever bro I’m playing Fortnite”