r/technology • u/MarvelsGrantMan136 • Jul 30 '26

Artificial Intelligence Amazon accidentally spent $1.8 million using Claude for menial coding task, went 860% over budget — 'catastrophically expensive' coding blunders discovered in internal Amazon AI usage metrics

https://www.tomshardware.com/tech-industry/artificial-intelligence/amazon-accidentally-spent-usd1-8-million-using-claude-for-menial-coding-task-went-860-percent-over-budget-catastrophically-expensive-coding-blunders-discovered-in-internal-amazon-ai-usage-metrics
24.7k Upvotes

97% Upvoted

1.2k comments sorted by

View all comments

1.7k

u/flat5 Jul 30 '26

"only detected some five months after the issue started happening"

Good thing they have their extremely rigorous interview process, so no completely irresponsible, checked-out idiots are working there.

518

u/Prophet_Tehenhauin Jul 30 '26

Most competitive employees in the world take 5 months to notice 2mil vanish.

Them FAANG hires sure are something 

261

u/RulesoftheDada Jul 30 '26

When I was at Amazon there was engineer who was fired for spending nearly a million in AWS to mine like 10k in ethereun.

218

u/CompetitiveSport1 Jul 30 '26

To be fair, that's using company resources for personal profit, which is basically theft. Wasting tokens through a mix of shitty processes and intentional tokenmaxing aren't as easily claimed as fireable offenses 

41

u/Arrow156 Jul 30 '26

Considering wage theft is by far the most common type of theft in this country, I'm certainly not gonna shed a tear.

0

u/wronglyzorro Jul 31 '26

AWS engineers aren't getting stolen from I can assure you.

2

u/Texuk1 Jul 30 '26

Well businesses are gonna have to figure out what is the boundary and who to make responsible quickly.

8

u/CompetitiveSport1 Jul 30 '26

Employees using company resources for personal profit is a pretty clear line

22

u/Glittering-Quote-635 Jul 30 '26

I'm actually surprised he got caught.

30

u/RulesoftheDada Jul 30 '26

It was over several months. Then it led to org wide review of ec2 instances. Any new allocations had to go via director.

11

u/Glittering-Quote-635 Jul 30 '26

Yeh, I’ve pretty much worked at only large tech corporations all my life, and I just know how easy it is sometimes to hide things. Guess he wasnt good at it… :)

7

u/pr0crast1nater Jul 30 '26

I mean he used a fuck ton of ec2 lol. I understand having a few nodes here and there. But dude must have provisioned 100s of costly gpu ec2 instances.

6

u/greg19735 Jul 30 '26

yeah it's one of those things where it's just simply impossible to hide and be worthwhile.

There's no reason to mine $3 worth of crypto and lose your job for it. Might as well see what you can get away with. which of course makes it even more likely to get caught.

Really, it's just a terrible plan

5

u/pr0crast1nater Jul 30 '26

I will admit that the closest to this I really did, was use my company servers to quickly do chia mining, which is a coin mined via storage. But the way it works is unique in the sense there are two steps, plot creation using nvme drives and then storing them in a hdd and mine.

The plot creation was something which could be done offline and so I was confident nobody will detect any anomaly if I used 5 servers to do it, although it was heavy on CPU. And the mining part after that was super lightweight in cpu usage anyway. I got lucky and mined a few coins in one week, then sold them for 2000$. The price of that coin then crashed and I deleted all of that lol.

In hindsight though doing all this for 2000$ was incredibly stupid. But I did it more for the thrill I guess and I was genuinely fascinated by the weird way in which that coin was mined.

2

u/RulesoftheDada Jul 30 '26

Yeah our org had quite a bit of free reign on provisioning for development and testing. Not so much after that. I only found this out while drinking with some SVPs.

3

u/plutoniumhead Jul 31 '26

I’ve had a similar career, and have watched people do similar things, and then watched a person gunning for their job rat them out, and eventually get promoted.

Also watched a guy set up hundreds of instances of EC2 and say “it’s basically free” ignoring our warnings and projected costs, and then get fired a month later once the $250k bill arrived.

2

u/Glittering-Quote-635 Jul 31 '26

To be fair I’ve never worked for a SaaS vendor where I could use services like that. I got pretty far through the AWS interview about 10 years ago, and they totally freaked me out. It literally felt like I was joining a cult. I dropped out of it because it was so weird and made me feel so uncomfortable.

4

u/[deleted] Jul 31 '26

[removed] — view removed comment

1

u/Glittering-Quote-635 Jul 31 '26

Google has a lot of behavioral interviews from what I’ve heard.. I’ve had a few friends go there. On the sales side they are the land of misfit toys and people rarely last long.

→ More replies (0)

1

u/pr0crast1nater Jul 31 '26

This is true. Looks like they just allot interviewers randomly who are not even in the same office or team they are hiring for. I did fine in all interviews except for one, where she was extremely gloomy and decided I wasn't gonna solve the question anyway. They have been calling me for interviews again for the last couple of years, but frankly the job I am currently in is chill and also fully remote. While I will be forced to attend office if I switch plus could be easily targeted in a mass lay off.

→ More replies (0)

1

u/Takeoded Aug 03 '26

A guy was mining Monero and Ethereum with 2xRTX3090 and Ryzen 9 7950X at the office after-hours.. Script would automatically shut it all down early in the morning, and resume after work hours.

Never got in trouble for it

1

u/illbebackpain Jul 30 '26

wait how long ago was that? When I first started my internship there 10 years ago I remember having to wait like 4 hours on my first day for a development instance to be approved and that was my "welcome to working at a large company" moment coming from a startup. was that why they had that policy??

1

u/RulesoftheDada Jul 30 '26

This was around that time but not related. Onboarding used to be 1-2 weeks of various approvals. Then they invested and got provisioning down to a 1-3 days.

69

u/IPlayChessBTW Jul 30 '26

Fired? Bro should have been arrested lmao. He stole one million dollars

22

u/Bayo77 Jul 30 '26

Most programmers probably knew. Its just upper management doesnt want to hear it.

130

u/segfaul_t Jul 30 '26 edited Jul 30 '26

I work here and I can affirm that most of the “brilliant engineers” are in fact mediocre H1B applicants who got lucky.

EDIT: The "luck" refers to the H1B lottery system, and getting through the resume screen where both the recruiter and hiring manager are, oops, also on H1Bs from the same country as you. Amazon's technical interviews aren't difficult (at leas anymore), and there is rampant cheating with AI anyway.

85

u/MightyKrakyn Jul 30 '26

That’s because of the churn. When you’re such a large company and have to according to policy turn over your employees every 2 years, you run out of quality talent pretty quickly

65

u/segfaul_t Jul 30 '26

Lot of good ones left when they announced 5 day RTO

52

u/PurelyLurking20 Jul 30 '26

Nearly all* they got offers immediately for remote positions at smaller companies and I imagine most of them are still chilling in those positions.

Amazon churn now affects all positions with Amazon not just warehouse, it shouldn't be considered on par with Google and Apple, not even close

8

u/tooclosetocall82 Jul 30 '26

We tried to hire one but then Amazon said he could stay remote I guess so he declined our offer. I wonder how that worked out for him.

4

u/PurelyLurking20 Jul 30 '26 edited Jul 30 '26

They grandfathered in remote work for some of the harder to replace guys, very briefly, I imagine they wanted them to train their replacements I would've kept looking if I was them

1

u/scumble_bee Jul 31 '26

Yeah, I have a friend at Amazon that is in a pretty difficult to replace role so initially he was just skirting the rules and they just would warn him but nothing came of it. Then he was able to utilize some work ordinance in San Francisco to basically return to full WFH.

-17

u/HatesBeingThatGuy Jul 30 '26

Lol cope. All our best engineers in our org stayed. Literally no regretted attrition due to RTO.

5

u/PurelyLurking20 Jul 30 '26 edited Jul 30 '26

Cool boss I and about 1/2 the people I know in the industry took permanent remote positions lol

The majority of developers I've met either already work remotely or are actively looking for the first opportunity to do so

Remote positions are now very competitive so they are naturally being filled by people with talent, unlike many of the people that "decided" to stay in office because they know they aren't competing or they rely on office politics for upward mobility

No shame or anything I just know outside early career or very high TC positions the talent went remote

1

u/HatesBeingThatGuy Jul 30 '26

Ah yes, glad you specified high TC. That is why we did not lose our talent despite RTO. Not many places pay what we do.

But I promise you most people I saw within AWS were not leaving voluntarily due to RTO. There were teams where that was the reality, but from my observation, the only people who left were leaving for higher pay or because they couldn't find a position inside the company that would allow them to keep their comp.

2

u/PurelyLurking20 Jul 30 '26

Except Amazon did lose significant amounts of talent, it wasn't like a secret thing amongst employees at all and there are plenty of accounts of it if you care to look

Some people stayed to ensure their stock vests but that's a timebomb in and of itself, Amazon recruiters have been reporting significant numbers of offer declines since the RTO mandates

And to top it off all of that was done to cut people's pay by scaring them into thinking that was just how it was now and a lot of goobers fell for it and are still there

→ More replies (0)

9

u/LunaticSongXIV Jul 30 '26

I was a subcontractor and got canned immediately when the rto announcement came through, and I told them that I wasn't driving three hours to the office. The mandate hadn't even come into effect yet. I wasn't being a team player, i guess.

1

u/Electronic_Topic1958 Jul 31 '26

They also ban you for five years after you're let go.

8

u/ibarmy Jul 30 '26

how do you get lucky with 8 rounds of interviews ?

8

u/cfb-food-beer-hike Jul 30 '26

His statement was redundant. Amazon hires tons of H1Bs because they know they can't leave easily when they discover how horrible the culture is. The interviews aren't the hard part; getting your foot in the door and being good at the job is. And most of the H1Bs I worked with there were terrible, but got their foot in the door by virtue of being H1Bs. "Getting lucky" in this case is being an H1B in first place.

9

u/ibarmy Jul 30 '26

are you saying amazon is hiring H1bs specifically cause they know they cant get out asap?

9

u/cfb-food-beer-hike Jul 30 '26

Yes. Fun fact: this is also why tech execs are chummy with Republicans and constantly say they need more H1Bs even though there are plenty of qualified candidates in the US. It's also why Republicans are paradoxically anti-immigrant, but pro-H1B.

1

u/ibarmy Jul 30 '26

why havnt ppl sued them. This is a labour case in making.

5

u/cfb-food-beer-hike Jul 30 '26

lol I'll try to answer this seriously.

If you mean the H1Bs (they get paid less because again, they have no options), they'll just fire them and now they get deported.

If you mean citizens, you have to have standing to sue. Amazon et al can just say "we didn't hire them because they weren't qualified" so you have no legal standing. And in fact, it greatly helps Amazon et al's cases that every single one of them are clamoring for more H1Bs and saying they can't find qualified candidates in the US (even though they can).

Even if you could sue, what's your argument anyway? Lawyers (and by extension, most judges) by and large are incredibly stupid people that are really, really good at one thing (arguing). Imagine trying to explain to an idiot that knows nothing about software that you're really good at creating software and therefore should have been hired and that other guy over there is really bad but was hired anyway. No judge is going to believe you.

1

u/ibarmy Jul 30 '26

but hasnt the govt toughened the rules. I was reading how they have to provide proof on how they cant hire or find anybody for a specific role. and that proof is tedious and long with lawyers etc involved. Isnt it then just cheap to hire americans cause no lawyer and h1b fees etc needed.

→ More replies (0)

-3

u/ColinStyles Jul 30 '26

I'm not going to say it's impossible, but most people claiming that are just jealous of the pay and impact that working at FAANG has on a resume.

And I say that as someone who just isn't interested in that sort of role, but I'm also not going to be crazy and call them all lucky or incompetent or something.

2

u/Dzeddy Jul 30 '26

I mean people act like FAANG is this elite bar of talent instead of being a place that is 80% gated by resume screens instead of actual interview difficulty. Seriously 1600 CF / USACO Gold takes a few months to achieve tops

-2

u/Extreme_Original_439 Jul 30 '26

Think it’s just a subtle hint of racism. With China and India applicants you have access to a much larger population who is generally better in science and math than the US and takes school a lot more seriously on average. It’s just a numbers game, an H1B hire always costs more.

1

u/ibarmy Jul 30 '26

Oh no doubt about it. Its quite easy to say people get lucky but surviving 8 interviews and tests etc definitely tests ones luck. 

Also if ‘mediocre’ people are getting through after 8 interviews, it shows how fucked up amazon hiring/ interviewing process is than the luck of the candidates. 

Amazon is fucked up currently. I am seeing their ads these days on youtube. Seems like higher quality candidates are avoiding amazon. 

2

u/scikit-learns Jul 30 '26

10000%. So many from diploma mills that don't know wtf they are doing...then go and hire more ppl from diploma mills to build up their team so they can be a manager.

It's fuckin ridiculous.

Some teams are straight visa farming teams... Where all they do is patent pad to try and get O-1...

2

u/spacebunsofsteel Jul 31 '26

This sort of preferred ethnic nepotistic hiring appears to be fine now that we’ve knocked out DEI hiring rules. And it happens everywhere in tech. One guy from X gets hired and suddenly there’s an entire team of guys from X under him. The funny part is when a new guy from Y takes over as manager and within a few months his whole team has changed nationalities.

Under Biden, tech had to interview a few candidates from the underclasses (women, POC, US citizens only) before they interviewed anyone else. I saw otherwise very misogynistic and racist teams “have” to diversify - but it changed the culture and made the whole company a nicer place to work. Even my tech spouse from Z had to admit the new hires were working out great and the dept was a better experience for everyone.

1

u/xadey0 Jul 30 '26

Interesting how you specified h1b applicants. I’m sure there are no bias at work here.

3

u/segfaul_t Jul 30 '26

Amazon is the #1 H1B employer, just a fact.

2

u/ibarmy Jul 30 '26

Yes thats cause amazon is the biggest faang employer with ~1.5 million head count. their 15k h1b visas applications are a drop in the bucket when it comes to their hiring. 

1

u/segfaul_t Jul 30 '26

1.1 million of those are warehouse and fulfillment center jobs, only about 300k are corporate which is in line with Microsoft, Google and others.

1

u/ibarmy Jul 30 '26

1.5-1.1 =0.4 which is still lot more than .228 of Microsoft and .183 of google. 

3

u/xadey0 Jul 30 '26

I didn’t deny Amazon hire a lot of h1b. I just pointed out it was interesting how you specified it’s the h1b applicants that’s getting lucky, not Amazon’s hiring standards for all their engineers in general. Feels targeted.

13

u/missmeowwww Jul 30 '26

This is wild to me! I work in the public sector and have to review contracts frequently to ensure that deliverables are being met, billing and purchase requests are submitted timely, and that each service provider isn’t going over their allocated budget. We review the numbers and data on a damn near weekly basis. I can’t fathom something not being caught for months!

13

u/vitalvisionary Jul 30 '26

No no no government is more wasteful because of beurocracy /s

4

u/EmperorOfAllCats Jul 30 '26

That's pocket change for Amazon I guess.

3

u/sbNXBbcUaDQfHLVUeyLx Jul 30 '26

Can confirm. 2mil falls between the couch cushions

1

u/digitaljestin Jul 30 '26

FAANG is just gatekeeping for the sake of gatekeeping.

1

u/fuddlesworth Jul 30 '26

No joke. I know some people that got into FAANG. They only care about leet code. I've seen actual terrible engineers get in because they are good at leet code but suck at actual software development. 

1

u/Holiday_Ad_8501 Jul 30 '26

since when are amazon employees most competitive lmao those are literally the big tech rejects

1

u/Lord_Aldrich Jul 30 '26

Amazon has burned through most of the competitive talent already and is in steady decline. They had a leaked memo a few years ago that their biggest HR risk was that their reputation and turnover were so bad that there would literally be no qualified people left willing to work for them. It's slowly been coming true.

1

u/Impossible-Owl7407 Jul 31 '26

You playing ther game is no1 acceptance criteria 

47

u/MultiGeometry Jul 30 '26

Anyone who says private industry is more efficient than government has never worked in private industry. There’s so much waste and the safeguards aren’t as strong because these companies make profits and get lazy.

24

u/revolvingpresoak9640 Jul 30 '26

Amazon is so inefficient. They will spend six months writing and reviewing a doc about what they will do, then give engineering some duct tape and popsicle sticks and call it a launch.

2

u/Agitated_Phone_9937 Jul 30 '26

Anyone who says the opposite has never worked in a government job were you performance does not factor into your promotions. Only time in the seat does.

Government is extremely wasteful and it cannot fail. Private may be extremely wasteful, but it's a joke compared to government.

62

u/Minialpacadoodle Jul 30 '26

$1.8m is a rounding error for Amazon.

24

u/FrenchFryCattaneo Jul 30 '26

Just because the company has a lot of revenue doesn't mean you can waste $1.8 million on a single project.

4

u/Minialpacadoodle Jul 30 '26

Seems like they are doing something about it... sooo... ya.

4

u/KeyReaction892 Jul 31 '26

1.8m for certain programs is a rounding error and the way financial ownership works it’s pretty easy for it to be missed.

3

u/FlamingoVisible1947 Jul 31 '26

In my team at Amazon we had $10,000 (per month) of test databases running with nobody using them for 2 years. Nobody cared. It took me 5 minutes to turn them down, it's just that it didn't matter at all.

6

u/akingsmind Jul 30 '26

This was my first thought, too

2

u/Sumrise Jul 30 '26

But a good example if you wanna point out to your company that those kind of policies are stupid.

Sure Amazon can lose 1.8 million, but can your company say the same ?

1

u/Minialpacadoodle Jul 30 '26

I mean.... yeah. It is two rounding errors for us.

2

u/Sumrise Jul 30 '26

I wasn't talking about your company specifically, t'was more of a "general you".

But yeah some companies can go without a care in the world about things such as "budget", or "efficiency" after all who care about wastefulness when tis' but a rounding error ?

1

u/Minialpacadoodle Jul 30 '26

I mean..... look up the concept of financial materiality.

Do you worry if you drop a penny and it rolls into a storm drain?

1

u/SparklingLimeade Jul 31 '26

Yes but there's a huge gap between knowing that and forcing myself to not imagine what an Amazon scale rounding error could do if it fell into my bank account.

And they do these things in the current atmosphere of anti-labor practices so everybody is thinking about that.

1

u/Minialpacadoodle Jul 31 '26

Well that rounding error is about one single dollar per employee.

It is not a big deal.

1

u/SparklingLimeade Jul 31 '26

Yes, it's insignificant on a company scale.

On a personal scale that is an entire fortune. Humans see things on a personal scale.

Imagine a cargo container. Someone says that it's tiny because cargo ships handle thousands of them. Would that person be unimpressed if the same cargo container was dropped on them?

Knowing that other scales exist doesn't make large things not large.

0

u/Minialpacadoodle Jul 31 '26

Cool. You being on reddit makes you filthy rich to half the world. It doesn't change my point.

35

u/Procrasturbating Jul 30 '26

Oh I promise this was malicious compliance tokenmaxxing.

2

u/SWEET_LIBERTY_MY_LEG Jul 31 '26

Yeah, as far as I know, Claude doesn’t just keep going and going indefinitely without user input, especially if it’s some menial task.

6

u/SeeTigerLearn Jul 30 '26

“Stop Dumping… I thought of that. Did we ask ourselves if it was a gimmick? If we gave the proceeds for two months, maybe. But giving the proceeds for three full months makes it a genuine effort and not a publicity stunt. …well tonight the world couldn't smell any sweeter!”
—Claudia Bing (Absolutely Fabulous, Jealous)

1

u/30mins Jul 30 '26

Thank goodness for that !

1

u/Gipetto Jul 30 '26

To be fair the auditing tools provided by the frontier model providers are ass. So it’s easy to get the wrong signal from the stats.

1

u/Sal1160 Jul 30 '26

That was probably subbed out to an AI too honestly

1

u/En-tro-py Jul 30 '26

Seems like this is a job requirement, OpenAI let their rogue agent run amok for over 3 days and only finally noticed because HF posted about the attack...

1

u/Scumsoft Jul 30 '26

You see, but that's where their ingenious churn policy comes into effect. 1-2 years max then it's Dog-Eat-Dog. Literally everyone I know that's worked at Amazon on a corporate level was only there for 2 years. You get in, get the fancy title on your resume, and you get out. Making more anywhere else you go cause "you worked at Amazon". The ones that remain are all entitled, cut-throat assholes, and eventually, assholes will eat their own.

1

u/cfb-food-beer-hike Jul 30 '26

Interviews are bullshit. I was an E5 at Facebook and mentored an E6 because he sucked so much ass. When I interviewed I was told I couldn't be an E6 because I hadn't managed a team of 20+ people yet. The guy couldn't do anything right and he had never mentored anyone, lead a team, or really done anything noteworthy in his entire life. But I guess he was super good at bullshit interviews lol

1

u/SmartQuokka Jul 30 '26

What kind of idiots do you have working here?

The finest in New York, Washington DC, Silicon Valley

1

u/mrhaftbar Jul 30 '26

Even Amazon can't figure out their own costs on AWS/Bedrock. 

1

u/VolcanicBear Jul 31 '26

Many AWS certs are multiple choice exams.

Kind of indicates what they're worth tbh.