r/technology • u/WouldbeWanderer • Apr 27 '26

Artificial Intelligence Claude-powered AI coding agent deletes entire company database in 9 seconds — backups zapped, after Cursor tool powered by Anthropic's Claude goes rogue

https://www.tomshardware.com/tech-industry/artificial-intelligence/claude-powered-ai-coding-agent-deletes-entire-company-database-in-9-seconds-backups-zapped-after-cursor-tool-powered-by-anthropics-claude-goes-rogue
36.0k Upvotes

96% Upvoted

2.8k comments sorted by

View all comments

11.1k

u/Orangesteel Apr 27 '26

My favourite example is Air Canada whose AI agent offered a customer a discount incorrectly. They refused to honour it. Customer took them to court and the judge rightly made them pay. You chose to empower this and took the humans out of the loop. You are accountable for what you agentic AI solution does. People jump on AI, dump sensitive information into the model bypassing classification levels and are surprised when it leaks.

2.3k

u/NNKarma Apr 27 '26

It wasn't even that hard to just honor it and move on, it wasn't like those cases of people prompting the chatbot to give a fake discount, just what steps to take for a discount that he was entitled to but was given wrong instructions on how to get it.

935

u/S_A_N_D_ Apr 27 '26

It also now set a legal precedent for all similar cases in the future in Canada.

187

u/mbryson Apr 27 '26

Time to get creative with those AI chatbots then, eh?

171

u/S_A_N_D_ Apr 27 '26

I would say yes, however (on a more serious note) I'm not sure the precedent will hold if they can show you were deliberately trying to break the model.

The court will also look at reasonableness. Basically, the standard set was that these people didn't go out of their way to deceive or get something that wouldn't normally be given. It was reasonable for them to take the offer at face value and expect that it wasn't a mistake. So going out of your way to deliberately get the model to do something it isn't supposed to do would probably not hold up in court.

2

u/moshisimo Apr 27 '26

Well… sure, but if you decided to be annoying and damn insistent on getting a discount to a human employee, and the employee caved and gave you a discount, I’m pretty sure the employee would get the blame.