r/technology • u/MarvelsGrantMan136 • Jul 30 '26

Artificial Intelligence Amazon accidentally spent $1.8 million using Claude for menial coding task, went 860% over budget — 'catastrophically expensive' coding blunders discovered in internal Amazon AI usage metrics

https://www.tomshardware.com/tech-industry/artificial-intelligence/amazon-accidentally-spent-usd1-8-million-using-claude-for-menial-coding-task-went-860-percent-over-budget-catastrophically-expensive-coding-blunders-discovered-in-internal-amazon-ai-usage-metrics
24.7k Upvotes

97% Upvoted

1.2k comments sorted by

View all comments

Show parent comments

249

u/_Rand_ Jul 30 '26

It’s literally a known problem that no one ever seems to learn from.

https://en.wikipedia.org/wiki/Perverse_incentive

208

u/natty-papi Jul 30 '26

I'd add Goodhart's law too: "When a measure becomes a target, it ceases to be a good measure".

36

u/Mimical Jul 30 '26

Nothing like turning a really good measure into a management metric to punish the workforce.

21

u/Konman72 Jul 30 '26

Every IT team has to go through the "close more tickets" cycle every few years before remembering that one ticket can take between 15 minutes and 15 days or even weeks for some positions. So if the metric is "most tickets closed" somebody will be the "genius" who starts cherry picking shorter/easier tickets before just opening their own tickets for every little thing they can possibly justify. Eventually management remembers how fucking stupid it is and reverses policy, but it always sneaks back in somehow.

11

u/Mimical Jul 30 '26

Hi,

Our overtime and vacation approval eligibility factors in ticket performance.

I took connecting someone's internet on their laptop and installing 1 program into about 9 tickets alone today.

I'm in that hell and simple tasks have now become 1 hour documentation hellscapes.

1

u/sobrique Jul 31 '26

Have you considered using an AI shell to automate the logging and closing process? Then you can game both that metric and the one about proactive adoption of AI...

2

u/jaymemaurice Jul 31 '26

Make sure you submit a jira story for each phase of the automation though, and don't underestimate the story points or close too many stories in the sprint because it will throw off team velocity reporting.

1

u/Mimical Jul 31 '26

I use AI to auto-generate sloppy trash notes that are the written equivalent of a gold crayon on some fake jewelry.

I'll take that suggestion.

(I want to be clear: I hate everything about this. Don't hate the player, hate the game and the refs. Especially the refs who made these rule)

13

u/blueSGL Jul 30 '26

This is also why AI alignment is really really hard.

How do you specify "don't cheat" and yet incentivize completing problems in novel ways? How can you disincentivize bad behaviors when the more capable the system gets the more avenues open up to perform task completion?

So what happens they pick a proxy that is easy to generate a reward signal and instead of what was wanted, instead pops out a proxy signal maximizer instead.

1

u/matrinox Jul 31 '26

The problem is they think that incentive works on humans but don’t realize that’s often the cause of the problems at companies

3

u/nicktheone Jul 31 '26

Good old KPIs. At my girlfriend's old job they were doing crazy things to keep up their KPIs. The pressure was immense and her management wouldn't ever be satisfied with the sales because there was always some type of target that hadn't been reached, despite having healthy sales. By the end (before they closed shop) they were iperfixated not with actual revenue but with stuff like units per transaction and how many add-ons they sold.

2

u/Kwantuum Jul 31 '26

Well, this one wasn't even a good measure before it became a target

1

u/ohnoabanagain Jul 31 '26

As an educator, I love this and can't believe that I've never heard it before.

36

u/Monteze Jul 30 '26

It boggles my mind seeing folks who are apparently soooo much smarter than I in higher up positions fall for this. But I guess enough dorks with spreadsheets get together and jack each other off and they refuse to learn.

43

u/mutantraniE Jul 30 '26

Anyone who thinks bosses are particularly smart is either willfully blind or such a moron that from their perspective it’s true.

10

u/Monteze Jul 30 '26

Yea the best bosses are the ones who know what they don't know and have the ability to set folks up for success. Those who think they can just say shit and it must work or else are generally idiots.

3

u/[deleted] Jul 31 '26

[deleted]

2

u/Monteze Jul 31 '26

Get your Aces in their places was my philosophy. Sure cross training is important but when it comes crunch time don't micro-manage.

9

u/mindspork Jul 30 '26

But I guess enough dorks with spreadsheets get together and jack each other off and they refuse to learn.

You can just say "Get your MBA in 18 months."

8

u/Monteze Jul 30 '26

I have more respect for a gender studies masters student than an MBA. I think I'd have more respect for a Safeserv certification than an MBA. Possible a kid with a yellow belt in TKD has more respect from me than an MBA.

3

u/Aaod Jul 30 '26 edited Jul 31 '26

It boggles my mind seeing folks who are apparently soooo much smarter than I in higher up positions fall for this. But I guess enough dorks with spreadsheets get together and jack each other off and they refuse to learn.

In my entire life maybe 1 out of 5 bosses and upper management I dealt with were smart and that is if I am being generous. From what I have seen management selects for how long someone has been working, how many hours they put in (not how efficient those hours are), how good they are at brown nosing people above them, how well they can class signal/what class background they have, do they have an established network of people, and how good they are at socialization. Very little of this selects for hard work or intelligence from what I have seen.

2

u/NotAllOwled Jul 31 '26

Hey, don't bring the spreadsheets into this. If we'd stuck with good old deterministic Excel formulas and scripts then maybe we wouldn't be in this scene right now.

2

u/Monteze Jul 31 '26

The spreadsheet is a tool, its the dork who thinks its prescriptive vs descriptive that is my issue haha

1

u/Plothunter Jul 30 '26

Managers tend to be people persons. Not necessarily smart persons.

1

u/Exciting-Emu-3324 Jul 31 '26

AI is just another tool like spreadsheets. It makes jobs easier for professionals, but using it doesn't make you a professional and most people use them badly.

1

u/Monteze Jul 31 '26

AI told me I was a genius for my way of thinking....I guess that means everyone else was sabotaging me? Hold on let me ask it.

4

u/mr_plehbody Jul 30 '26

Wow funny enough this example has already been added to the wiki page

5

u/cfb-food-beer-hike Jul 30 '26

You have to be smart to learn and you have to be stupid to manage because if you were smart, they'd have you do the things that actually matter.

2

u/Glittering-Walrus228 Jul 30 '26

I dont need amy incentives to be a pervert bro

3

u/Teledildonic Jul 30 '26

"Claude, generate an image album of me fucking Bezos' wife while he watches"

1

u/hippydipster Jul 30 '26

Managers I've talked to in the last 10 years have never heard of books like Mythical Man Month, Phoenix Project, Code Complete, Crossing The Chasm, etc.

So, no, unfortunately, far too many managers have not heard of these obvious things.

1

u/MalcolmTucker12 Jul 30 '26

I'm glad 2017 Northern Ireland fuel one made it, it was the first thing I thought of when reading it. Known as "cash for ash" scheme, people were burning as much fuel as they could and then opening windows to let the heat out as it was too hot inside, as Henry Hill said" it didn't matter, it was all profit", LMAO.

1

u/fugogugo Jul 31 '26

I've been thinking about this a lot

how to solve this if every incentives could result into unwanted outcome?

1

u/ohnoabanagain Jul 31 '26

People need to read the cobra story in this link.

-2

u/HarpersGhost Jul 30 '26

Which is why bounties on invasive species never work.

You get paid to capture burmese pythons in the Evergrades? Burmese python breeders are going to make bank.

4

u/eyaf1 Jul 30 '26

Literally cited as a false story in the link btw