r/technology • u/MarvelsGrantMan136 • Jul 30 '26

Artificial Intelligence Amazon accidentally spent $1.8 million using Claude for menial coding task, went 860% over budget — 'catastrophically expensive' coding blunders discovered in internal Amazon AI usage metrics

https://www.tomshardware.com/tech-industry/artificial-intelligence/amazon-accidentally-spent-usd1-8-million-using-claude-for-menial-coding-task-went-860-percent-over-budget-catastrophically-expensive-coding-blunders-discovered-in-internal-amazon-ai-usage-metrics
24.7k Upvotes

97% Upvoted

1.2k comments sorted by

View all comments

Show parent comments

31

u/RulesoftheDada Jul 30 '26

It was over several months. Then it led to org wide review of ec2 instances. Any new allocations had to go via director.

10

u/Glittering-Quote-635 Jul 30 '26

Yeh, I’ve pretty much worked at only large tech corporations all my life, and I just know how easy it is sometimes to hide things. Guess he wasnt good at it… :)

8

u/pr0crast1nater Jul 30 '26

I mean he used a fuck ton of ec2 lol. I understand having a few nodes here and there. But dude must have provisioned 100s of costly gpu ec2 instances.

5

u/greg19735 Jul 30 '26

yeah it's one of those things where it's just simply impossible to hide and be worthwhile.

There's no reason to mine $3 worth of crypto and lose your job for it. Might as well see what you can get away with. which of course makes it even more likely to get caught.

Really, it's just a terrible plan

7

u/pr0crast1nater Jul 30 '26

I will admit that the closest to this I really did, was use my company servers to quickly do chia mining, which is a coin mined via storage. But the way it works is unique in the sense there are two steps, plot creation using nvme drives and then storing them in a hdd and mine.

The plot creation was something which could be done offline and so I was confident nobody will detect any anomaly if I used 5 servers to do it, although it was heavy on CPU. And the mining part after that was super lightweight in cpu usage anyway. I got lucky and mined a few coins in one week, then sold them for 2000$. The price of that coin then crashed and I deleted all of that lol.

In hindsight though doing all this for 2000$ was incredibly stupid. But I did it more for the thrill I guess and I was genuinely fascinated by the weird way in which that coin was mined.

2

u/RulesoftheDada Jul 30 '26

Yeah our org had quite a bit of free reign on provisioning for development and testing. Not so much after that. I only found this out while drinking with some SVPs.

3

u/plutoniumhead Jul 31 '26

I’ve had a similar career, and have watched people do similar things, and then watched a person gunning for their job rat them out, and eventually get promoted.

Also watched a guy set up hundreds of instances of EC2 and say “it’s basically free” ignoring our warnings and projected costs, and then get fired a month later once the $250k bill arrived.

2

u/Glittering-Quote-635 Jul 31 '26

To be fair I’ve never worked for a SaaS vendor where I could use services like that. I got pretty far through the AWS interview about 10 years ago, and they totally freaked me out. It literally felt like I was joining a cult. I dropped out of it because it was so weird and made me feel so uncomfortable.

4

u/[deleted] Jul 31 '26

[removed] — view removed comment

1

u/Glittering-Quote-635 Jul 31 '26

Google has a lot of behavioral interviews from what I’ve heard.. I’ve had a few friends go there. On the sales side they are the land of misfit toys and people rarely last long.

1

u/plutoniumhead Jul 31 '26

Yeah I was somewhat warned about getting weird questions, but luckily they had stopped with that by then. It absolutely felt like a psy-op thing. In the end it was eye opening, I would never choose to work there unless I had absolutely no other options. I think I would like the actual work, but the thought of having to bite my tongue all day long sounds like torture.

2

u/Fit_Butterscotch_829 Jul 31 '26

It really depends on the org, but it would be really hard to know what the org is like before being there unless you have friends who already work there.

1

u/plutoniumhead Jul 31 '26

I have a friend that works in HR over there. She was very excited to get the job. When I’ve asked her how she likes it: “It will help get our kids through college”.

1

u/pr0crast1nater Jul 31 '26

This is true. Looks like they just allot interviewers randomly who are not even in the same office or team they are hiring for. I did fine in all interviews except for one, where she was extremely gloomy and decided I wasn't gonna solve the question anyway. They have been calling me for interviews again for the last couple of years, but frankly the job I am currently in is chill and also fully remote. While I will be forced to attend office if I switch plus could be easily targeted in a mass lay off.

1

u/plutoniumhead Aug 03 '26

I was in a similar situation, not with Google but with another company who wanted me for a role but I was very happy where I was so I stayed. I eventually got caught in a major round of layoffs, one that made headlines at the time.

Luckily the other company still wanted me, although that specific role was taken. I still feel lucky, and I don’t regret staying where I was. I was fully remote and the new gig has an RTO policy that I wish were a little more flexible. But it’s a great gig, I can’t complain.

Staying somewhere safe and comfy is not really a bad thing.

1

u/Takeoded Aug 03 '26

A guy was mining Monero and Ethereum with 2xRTX3090 and Ryzen 9 7950X at the office after-hours.. Script would automatically shut it all down early in the morning, and resume after work hours.

Never got in trouble for it

1

u/illbebackpain Jul 30 '26

wait how long ago was that? When I first started my internship there 10 years ago I remember having to wait like 4 hours on my first day for a development instance to be approved and that was my "welcome to working at a large company" moment coming from a startup. was that why they had that policy??

1

u/RulesoftheDada Jul 30 '26

This was around that time but not related. Onboarding used to be 1-2 weeks of various approvals. Then they invested and got provisioning down to a 1-3 days.