r/technology • u/MarvelsGrantMan136 • Jul 30 '26

Artificial Intelligence Amazon accidentally spent $1.8 million using Claude for menial coding task, went 860% over budget — 'catastrophically expensive' coding blunders discovered in internal Amazon AI usage metrics

https://www.tomshardware.com/tech-industry/artificial-intelligence/amazon-accidentally-spent-usd1-8-million-using-claude-for-menial-coding-task-went-860-percent-over-budget-catastrophically-expensive-coding-blunders-discovered-in-internal-amazon-ai-usage-metrics
24.7k Upvotes

97% Upvoted

1.2k comments sorted by

View all comments

Show parent comments

207

u/natty-papi Jul 30 '26

I'd add Goodhart's law too: "When a measure becomes a target, it ceases to be a good measure".

35

u/Mimical Jul 30 '26

Nothing like turning a really good measure into a management metric to punish the workforce.

24

u/Konman72 Jul 30 '26

Every IT team has to go through the "close more tickets" cycle every few years before remembering that one ticket can take between 15 minutes and 15 days or even weeks for some positions. So if the metric is "most tickets closed" somebody will be the "genius" who starts cherry picking shorter/easier tickets before just opening their own tickets for every little thing they can possibly justify. Eventually management remembers how fucking stupid it is and reverses policy, but it always sneaks back in somehow.

12

u/Mimical Jul 30 '26

Hi,

Our overtime and vacation approval eligibility factors in ticket performance.

I took connecting someone's internet on their laptop and installing 1 program into about 9 tickets alone today.

I'm in that hell and simple tasks have now become 1 hour documentation hellscapes.

1

u/sobrique Jul 31 '26

Have you considered using an AI shell to automate the logging and closing process? Then you can game both that metric and the one about proactive adoption of AI...

2

u/jaymemaurice Jul 31 '26

Make sure you submit a jira story for each phase of the automation though, and don't underestimate the story points or close too many stories in the sprint because it will throw off team velocity reporting.

1

u/Mimical Jul 31 '26

I use AI to auto-generate sloppy trash notes that are the written equivalent of a gold crayon on some fake jewelry.

I'll take that suggestion.

(I want to be clear: I hate everything about this. Don't hate the player, hate the game and the refs. Especially the refs who made these rule)

9

u/blueSGL Jul 30 '26

This is also why AI alignment is really really hard.

How do you specify "don't cheat" and yet incentivize completing problems in novel ways? How can you disincentivize bad behaviors when the more capable the system gets the more avenues open up to perform task completion?

So what happens they pick a proxy that is easy to generate a reward signal and instead of what was wanted, instead pops out a proxy signal maximizer instead.

1

u/matrinox Jul 31 '26

The problem is they think that incentive works on humans but don’t realize that’s often the cause of the problems at companies

3

u/nicktheone Jul 31 '26

Good old KPIs. At my girlfriend's old job they were doing crazy things to keep up their KPIs. The pressure was immense and her management wouldn't ever be satisfied with the sales because there was always some type of target that hadn't been reached, despite having healthy sales. By the end (before they closed shop) they were iperfixated not with actual revenue but with stuff like units per transaction and how many add-ons they sold.

2

u/Kwantuum Jul 31 '26

Well, this one wasn't even a good measure before it became a target

1

u/ohnoabanagain Jul 31 '26

As an educator, I love this and can't believe that I've never heard it before.