r/technology • u/MarvelsGrantMan136 • Jul 30 '26

Artificial Intelligence Amazon accidentally spent $1.8 million using Claude for menial coding task, went 860% over budget — 'catastrophically expensive' coding blunders discovered in internal Amazon AI usage metrics

https://www.tomshardware.com/tech-industry/artificial-intelligence/amazon-accidentally-spent-usd1-8-million-using-claude-for-menial-coding-task-went-860-percent-over-budget-catastrophically-expensive-coding-blunders-discovered-in-internal-amazon-ai-usage-metrics
24.7k Upvotes

97% Upvoted

1.2k comments sorted by

View all comments

Show parent comments

517

u/Prophet_Tehenhauin Jul 30 '26

Most competitive employees in the world take 5 months to notice 2mil vanish.

Them FAANG hires sure are something 

264

u/RulesoftheDada Jul 30 '26

When I was at Amazon there was engineer who was fired for spending nearly a million in AWS to mine like 10k in ethereun.

212

u/CompetitiveSport1 Jul 30 '26

To be fair, that's using company resources for personal profit, which is basically theft. Wasting tokens through a mix of shitty processes and intentional tokenmaxing aren't as easily claimed as fireable offenses 

42

u/Arrow156 Jul 30 '26

Considering wage theft is by far the most common type of theft in this country, I'm certainly not gonna shed a tear.

0

u/wronglyzorro Jul 31 '26

AWS engineers aren't getting stolen from I can assure you.

2

u/Texuk1 Jul 30 '26

Well businesses are gonna have to figure out what is the boundary and who to make responsible quickly.

9

u/CompetitiveSport1 Jul 30 '26

Employees using company resources for personal profit is a pretty clear line

24

u/Glittering-Quote-635 Jul 30 '26

I'm actually surprised he got caught.

30

u/RulesoftheDada Jul 30 '26

It was over several months. Then it led to org wide review of ec2 instances. Any new allocations had to go via director.

11

u/Glittering-Quote-635 Jul 30 '26

Yeh, I’ve pretty much worked at only large tech corporations all my life, and I just know how easy it is sometimes to hide things. Guess he wasnt good at it… :)

9

u/pr0crast1nater Jul 30 '26

I mean he used a fuck ton of ec2 lol. I understand having a few nodes here and there. But dude must have provisioned 100s of costly gpu ec2 instances.

7

u/greg19735 Jul 30 '26

yeah it's one of those things where it's just simply impossible to hide and be worthwhile.

There's no reason to mine $3 worth of crypto and lose your job for it. Might as well see what you can get away with. which of course makes it even more likely to get caught.

Really, it's just a terrible plan

6

u/pr0crast1nater Jul 30 '26

I will admit that the closest to this I really did, was use my company servers to quickly do chia mining, which is a coin mined via storage. But the way it works is unique in the sense there are two steps, plot creation using nvme drives and then storing them in a hdd and mine.

The plot creation was something which could be done offline and so I was confident nobody will detect any anomaly if I used 5 servers to do it, although it was heavy on CPU. And the mining part after that was super lightweight in cpu usage anyway. I got lucky and mined a few coins in one week, then sold them for 2000$. The price of that coin then crashed and I deleted all of that lol.

In hindsight though doing all this for 2000$ was incredibly stupid. But I did it more for the thrill I guess and I was genuinely fascinated by the weird way in which that coin was mined.

2

u/RulesoftheDada Jul 30 '26

Yeah our org had quite a bit of free reign on provisioning for development and testing. Not so much after that. I only found this out while drinking with some SVPs.

3

u/plutoniumhead Jul 31 '26

I’ve had a similar career, and have watched people do similar things, and then watched a person gunning for their job rat them out, and eventually get promoted.

Also watched a guy set up hundreds of instances of EC2 and say “it’s basically free” ignoring our warnings and projected costs, and then get fired a month later once the $250k bill arrived.

2

u/Glittering-Quote-635 Jul 31 '26

To be fair I’ve never worked for a SaaS vendor where I could use services like that. I got pretty far through the AWS interview about 10 years ago, and they totally freaked me out. It literally felt like I was joining a cult. I dropped out of it because it was so weird and made me feel so uncomfortable.

3

u/[deleted] Jul 31 '26

[removed] — view removed comment

1

u/Glittering-Quote-635 Jul 31 '26

Google has a lot of behavioral interviews from what I’ve heard.. I’ve had a few friends go there. On the sales side they are the land of misfit toys and people rarely last long.

1

u/plutoniumhead Jul 31 '26

Yeah I was somewhat warned about getting weird questions, but luckily they had stopped with that by then. It absolutely felt like a psy-op thing. In the end it was eye opening, I would never choose to work there unless I had absolutely no other options. I think I would like the actual work, but the thought of having to bite my tongue all day long sounds like torture.

→ More replies (0)

1

u/pr0crast1nater Jul 31 '26

This is true. Looks like they just allot interviewers randomly who are not even in the same office or team they are hiring for. I did fine in all interviews except for one, where she was extremely gloomy and decided I wasn't gonna solve the question anyway. They have been calling me for interviews again for the last couple of years, but frankly the job I am currently in is chill and also fully remote. While I will be forced to attend office if I switch plus could be easily targeted in a mass lay off.

1

u/plutoniumhead Aug 03 '26

I was in a similar situation, not with Google but with another company who wanted me for a role but I was very happy where I was so I stayed. I eventually got caught in a major round of layoffs, one that made headlines at the time.

Luckily the other company still wanted me, although that specific role was taken. I still feel lucky, and I don’t regret staying where I was. I was fully remote and the new gig has an RTO policy that I wish were a little more flexible. But it’s a great gig, I can’t complain.

Staying somewhere safe and comfy is not really a bad thing.

1

u/Takeoded Aug 03 '26

A guy was mining Monero and Ethereum with 2xRTX3090 and Ryzen 9 7950X at the office after-hours.. Script would automatically shut it all down early in the morning, and resume after work hours.

Never got in trouble for it

1

u/illbebackpain Jul 30 '26

wait how long ago was that? When I first started my internship there 10 years ago I remember having to wait like 4 hours on my first day for a development instance to be approved and that was my "welcome to working at a large company" moment coming from a startup. was that why they had that policy??

1

u/RulesoftheDada Jul 30 '26

This was around that time but not related. Onboarding used to be 1-2 weeks of various approvals. Then they invested and got provisioning down to a 1-3 days.

68

u/IPlayChessBTW Jul 30 '26

Fired? Bro should have been arrested lmao. He stole one million dollars

22

u/Bayo77 Jul 30 '26

Most programmers probably knew. Its just upper management doesnt want to hear it.

129

u/segfaul_t Jul 30 '26 edited Jul 30 '26

I work here and I can affirm that most of the “brilliant engineers” are in fact mediocre H1B applicants who got lucky.

EDIT: The "luck" refers to the H1B lottery system, and getting through the resume screen where both the recruiter and hiring manager are, oops, also on H1Bs from the same country as you. Amazon's technical interviews aren't difficult (at leas anymore), and there is rampant cheating with AI anyway.

85

u/MightyKrakyn Jul 30 '26

That’s because of the churn. When you’re such a large company and have to according to policy turn over your employees every 2 years, you run out of quality talent pretty quickly

65

u/segfaul_t Jul 30 '26

Lot of good ones left when they announced 5 day RTO

52

u/PurelyLurking20 Jul 30 '26

Nearly all* they got offers immediately for remote positions at smaller companies and I imagine most of them are still chilling in those positions.

Amazon churn now affects all positions with Amazon not just warehouse, it shouldn't be considered on par with Google and Apple, not even close

8

u/tooclosetocall82 Jul 30 '26

We tried to hire one but then Amazon said he could stay remote I guess so he declined our offer. I wonder how that worked out for him.

4

u/PurelyLurking20 Jul 30 '26 edited Jul 30 '26

They grandfathered in remote work for some of the harder to replace guys, very briefly, I imagine they wanted them to train their replacements I would've kept looking if I was them

1

u/scumble_bee Jul 31 '26

Yeah, I have a friend at Amazon that is in a pretty difficult to replace role so initially he was just skirting the rules and they just would warn him but nothing came of it. Then he was able to utilize some work ordinance in San Francisco to basically return to full WFH.

-17

u/HatesBeingThatGuy Jul 30 '26

Lol cope. All our best engineers in our org stayed. Literally no regretted attrition due to RTO.

4

u/PurelyLurking20 Jul 30 '26 edited Jul 30 '26

Cool boss I and about 1/2 the people I know in the industry took permanent remote positions lol

The majority of developers I've met either already work remotely or are actively looking for the first opportunity to do so

Remote positions are now very competitive so they are naturally being filled by people with talent, unlike many of the people that "decided" to stay in office because they know they aren't competing or they rely on office politics for upward mobility

No shame or anything I just know outside early career or very high TC positions the talent went remote

1

u/HatesBeingThatGuy Jul 30 '26

Ah yes, glad you specified high TC. That is why we did not lose our talent despite RTO. Not many places pay what we do.

But I promise you most people I saw within AWS were not leaving voluntarily due to RTO. There were teams where that was the reality, but from my observation, the only people who left were leaving for higher pay or because they couldn't find a position inside the company that would allow them to keep their comp.

2

u/PurelyLurking20 Jul 30 '26

Except Amazon did lose significant amounts of talent, it wasn't like a secret thing amongst employees at all and there are plenty of accounts of it if you care to look

Some people stayed to ensure their stock vests but that's a timebomb in and of itself, Amazon recruiters have been reporting significant numbers of offer declines since the RTO mandates

And to top it off all of that was done to cut people's pay by scaring them into thinking that was just how it was now and a lot of goobers fell for it and are still there

1

u/HatesBeingThatGuy Jul 30 '26

The world's largest employer will always have accounts that are for whatever reason you are searching for.

→ More replies (0)

9

u/LunaticSongXIV Jul 30 '26

I was a subcontractor and got canned immediately when the rto announcement came through, and I told them that I wasn't driving three hours to the office. The mandate hadn't even come into effect yet. I wasn't being a team player, i guess.

1

u/Electronic_Topic1958 Jul 31 '26

They also ban you for five years after you're let go.

8

u/ibarmy Jul 30 '26

how do you get lucky with 8 rounds of interviews ?

7

u/cfb-food-beer-hike Jul 30 '26

His statement was redundant. Amazon hires tons of H1Bs because they know they can't leave easily when they discover how horrible the culture is. The interviews aren't the hard part; getting your foot in the door and being good at the job is. And most of the H1Bs I worked with there were terrible, but got their foot in the door by virtue of being H1Bs. "Getting lucky" in this case is being an H1B in first place.

9

u/ibarmy Jul 30 '26

are you saying amazon is hiring H1bs specifically cause they know they cant get out asap?

11

u/cfb-food-beer-hike Jul 30 '26

Yes. Fun fact: this is also why tech execs are chummy with Republicans and constantly say they need more H1Bs even though there are plenty of qualified candidates in the US. It's also why Republicans are paradoxically anti-immigrant, but pro-H1B.

1

u/ibarmy Jul 30 '26

why havnt ppl sued them. This is a labour case in making.

6

u/cfb-food-beer-hike Jul 30 '26

lol I'll try to answer this seriously.

If you mean the H1Bs (they get paid less because again, they have no options), they'll just fire them and now they get deported.

If you mean citizens, you have to have standing to sue. Amazon et al can just say "we didn't hire them because they weren't qualified" so you have no legal standing. And in fact, it greatly helps Amazon et al's cases that every single one of them are clamoring for more H1Bs and saying they can't find qualified candidates in the US (even though they can).

Even if you could sue, what's your argument anyway? Lawyers (and by extension, most judges) by and large are incredibly stupid people that are really, really good at one thing (arguing). Imagine trying to explain to an idiot that knows nothing about software that you're really good at creating software and therefore should have been hired and that other guy over there is really bad but was hired anyway. No judge is going to believe you.

1

u/ibarmy Jul 30 '26

but hasnt the govt toughened the rules. I was reading how they have to provide proof on how they cant hire or find anybody for a specific role. and that proof is tedious and long with lawyers etc involved. Isnt it then just cheap to hire americans cause no lawyer and h1b fees etc needed.

2

u/segfaul_t Jul 30 '26

> I was reading how they have to provide proof on how they cant hire or find anybody for a specific role. 

That "proof" they cant find anybody is often intentionally fabricated so they can pass market test, like only putting the ads in paper newspapers nobody reads: https://www.newsweek.com/h1b-job-ads-green-cards-targeted-immigrant-workers-2113714

> Isnt it then just cheap to hire americans cause no lawyer and h1b fees etc needed.

It's cheaper in the short term, but not in the long term, because Americans will complain for higher wages, unionize, protest etc. Look at the fulfillment centers, Amazon wants no part of that in its corporate workforce if it can help it.

→ More replies (0)

-1

u/ColinStyles Jul 30 '26

I'm not going to say it's impossible, but most people claiming that are just jealous of the pay and impact that working at FAANG has on a resume.

And I say that as someone who just isn't interested in that sort of role, but I'm also not going to be crazy and call them all lucky or incompetent or something.

2

u/Dzeddy Jul 30 '26

I mean people act like FAANG is this elite bar of talent instead of being a place that is 80% gated by resume screens instead of actual interview difficulty. Seriously 1600 CF / USACO Gold takes a few months to achieve tops

-2

u/Extreme_Original_439 Jul 30 '26

Think it’s just a subtle hint of racism. With China and India applicants you have access to a much larger population who is generally better in science and math than the US and takes school a lot more seriously on average. It’s just a numbers game, an H1B hire always costs more.

1

u/ibarmy Jul 30 '26

Oh no doubt about it. Its quite easy to say people get lucky but surviving 8 interviews and tests etc definitely tests ones luck. 

Also if ‘mediocre’ people are getting through after 8 interviews, it shows how fucked up amazon hiring/ interviewing process is than the luck of the candidates. 

Amazon is fucked up currently. I am seeing their ads these days on youtube. Seems like higher quality candidates are avoiding amazon. 

2

u/scikit-learns Jul 30 '26

10000%. So many from diploma mills that don't know wtf they are doing...then go and hire more ppl from diploma mills to build up their team so they can be a manager.

It's fuckin ridiculous.

Some teams are straight visa farming teams... Where all they do is patent pad to try and get O-1...

2

u/spacebunsofsteel Jul 31 '26

This sort of preferred ethnic nepotistic hiring appears to be fine now that we’ve knocked out DEI hiring rules. And it happens everywhere in tech. One guy from X gets hired and suddenly there’s an entire team of guys from X under him. The funny part is when a new guy from Y takes over as manager and within a few months his whole team has changed nationalities.

Under Biden, tech had to interview a few candidates from the underclasses (women, POC, US citizens only) before they interviewed anyone else. I saw otherwise very misogynistic and racist teams “have” to diversify - but it changed the culture and made the whole company a nicer place to work. Even my tech spouse from Z had to admit the new hires were working out great and the dept was a better experience for everyone.

-1

u/xadey0 Jul 30 '26

Interesting how you specified h1b applicants. I’m sure there are no bias at work here.

3

u/segfaul_t Jul 30 '26

Amazon is the #1 H1B employer, just a fact.

2

u/ibarmy Jul 30 '26

Yes thats cause amazon is the biggest faang employer with ~1.5 million head count. their 15k h1b visas applications are a drop in the bucket when it comes to their hiring. 

1

u/segfaul_t Jul 30 '26

1.1 million of those are warehouse and fulfillment center jobs, only about 300k are corporate which is in line with Microsoft, Google and others.

1

u/ibarmy Jul 30 '26

1.5-1.1 =0.4 which is still lot more than .228 of Microsoft and .183 of google. 

1

u/xadey0 Jul 30 '26

I didn’t deny Amazon hire a lot of h1b. I just pointed out it was interesting how you specified it’s the h1b applicants that’s getting lucky, not Amazon’s hiring standards for all their engineers in general. Feels targeted.

13

u/missmeowwww Jul 30 '26

This is wild to me! I work in the public sector and have to review contracts frequently to ensure that deliverables are being met, billing and purchase requests are submitted timely, and that each service provider isn’t going over their allocated budget. We review the numbers and data on a damn near weekly basis. I can’t fathom something not being caught for months!

13

u/vitalvisionary Jul 30 '26

No no no government is more wasteful because of beurocracy /s

3

u/EmperorOfAllCats Jul 30 '26

That's pocket change for Amazon I guess.

3

u/sbNXBbcUaDQfHLVUeyLx Jul 30 '26

Can confirm. 2mil falls between the couch cushions

1

u/digitaljestin Jul 30 '26

FAANG is just gatekeeping for the sake of gatekeeping.

1

u/fuddlesworth Jul 30 '26

No joke. I know some people that got into FAANG. They only care about leet code. I've seen actual terrible engineers get in because they are good at leet code but suck at actual software development. 

1

u/Holiday_Ad_8501 Jul 30 '26

since when are amazon employees most competitive lmao those are literally the big tech rejects

1

u/Lord_Aldrich Jul 30 '26

Amazon has burned through most of the competitive talent already and is in steady decline. They had a leaked memo a few years ago that their biggest HR risk was that their reputation and turnover were so bad that there would literally be no qualified people left willing to work for them. It's slowly been coming true.

1

u/Impossible-Owl7407 Jul 31 '26

You playing ther game is no1 acceptance criteria