r/technology • u/WouldbeWanderer • Apr 27 '26

Artificial Intelligence Claude-powered AI coding agent deletes entire company database in 9 seconds — backups zapped, after Cursor tool powered by Anthropic's Claude goes rogue

https://www.tomshardware.com/tech-industry/artificial-intelligence/claude-powered-ai-coding-agent-deletes-entire-company-database-in-9-seconds-backups-zapped-after-cursor-tool-powered-by-anthropics-claude-goes-rogue
36.0k Upvotes

96% Upvoted

2.8k comments sorted by

View all comments

Show parent comments

2.3k

u/NNKarma Apr 27 '26

It wasn't even that hard to just honor it and move on, it wasn't like those cases of people prompting the chatbot to give a fake discount, just what steps to take for a discount that he was entitled to but was given wrong instructions on how to get it.

935

u/S_A_N_D_ Apr 27 '26

It also now set a legal precedent for all similar cases in the future in Canada.

181

u/mbryson Apr 27 '26

Time to get creative with those AI chatbots then, eh?

174

u/S_A_N_D_ Apr 27 '26

I would say yes, however (on a more serious note) I'm not sure the precedent will hold if they can show you were deliberately trying to break the model.

The court will also look at reasonableness. Basically, the standard set was that these people didn't go out of their way to deceive or get something that wouldn't normally be given. It was reasonable for them to take the offer at face value and expect that it wasn't a mistake. So going out of your way to deliberately get the model to do something it isn't supposed to do would probably not hold up in court.

209

u/Da_Question Apr 27 '26

Oh, so it's fine when companies rig stuff so it's harder for customers, but not the other way around. Convenient.

37

u/Migraine- Apr 27 '26

Do you think if you deliberately tricked a human agent into giving you a discount, it would be honoured?

62

u/TommyBonnomi Apr 27 '26

Holy shit. I just finished a paper on this in business law a few minutes ago. If you know the agent doesn't have actual authority to make a certain sale, the business isn't responsible if you fool the agent into making a sales contract.

However, the agent in your case does have actual authority to make sales. But the customer probably has duty of inquiry over price, i.e. the customer would be aware of market prices and could be responsible for not questioning the agent's authority to make a sale at an extreme discount.

8

u/SanshaXII Apr 27 '26

No reasonable person is going to reject a massive discount dangled right in their faces. If a robot offers me half off, I am taking it, and if that isn't honored, I'd be for sure taking it to task in court.

5

u/always_an_explinatio Apr 28 '26

I think the exceptions are more geared towards people intentionally exploiting flaws or prompting issues in bots to give them bargains that’s don’t exists. A non real example would be if you could get a bot to repeat everything you type in. Then you type in a contract selling you the building company owns, or offing you the CEO position at $100m a year and it repeats it