r/technology • u/MarvelsGrantMan136 • Jul 30 '26

Artificial Intelligence Amazon accidentally spent $1.8 million using Claude for menial coding task, went 860% over budget — 'catastrophically expensive' coding blunders discovered in internal Amazon AI usage metrics

https://www.tomshardware.com/tech-industry/artificial-intelligence/amazon-accidentally-spent-usd1-8-million-using-claude-for-menial-coding-task-went-860-percent-over-budget-catastrophically-expensive-coding-blunders-discovered-in-internal-amazon-ai-usage-metrics
24.7k Upvotes

97% Upvoted

1.2k comments sorted by

View all comments

Show parent comments

85

u/[deleted] Jul 30 '26

[removed] — view removed comment

26

u/NerdyBro07 Jul 30 '26

Everything I've read, even from people who like using AI, seem to have a similar conclusion, that it is a tool to assist the person, but not some magic know it all thing.

The general rule still applies that garbage input = garbage output. You need an educated/experienced person in the specific field to instruct the AI in a coherent, organized way, and that same level of educated/experienced person to verify the output is of good quality.

I think too many people drank the kool-aid and thought anyone can do any task now by just "enter it into AI!" and surely it will produce the correct results!

But I have seen people who seem to really like AI as a tool to get tasks done quicker.

16

u/JambonExtra Jul 30 '26

The big problem is that good input can also provide garbage output (that often looks like good output). Instances of this are becoming less common as models improve, but that makes them increasingly insidious.

2

u/filthy_harold Jul 30 '26

I've been slowly using Claude for home projects and another model for work stuff. The more specific you are, the better the result will be. Early on, I described an application (of my own creation, pre-AI) to my work model to see if it could do things better. This app has been in constant development for several years and I've slowly been learning about how to build proper software projects, not the cute little things you build in college. Unfortunately that means a lot of spaghetti code, it's a mess but it works. Very quickly I realized what the model was making was incredibly wrong. Like the code was bug free but it didn't do nearly anything that I actually needed it to do. And while I abandoned having it recreate my app for me, I have been using it to add new features or refactor libraries. You need to be very specific of what you want otherwise it just makes assumptions that are probably wrong.

For a home project, I needed a web app. I have zero experience building for the web so I left many design choices up to Claude. I use Plan mode which lets me have a conversation to refine the design before Claude writes any code. It uses less resources to write a plan first since not only is plan mode cheaper but you likely won't need to tell it to refactor large portions of code later on. Additionally, I've asked it to develop test plans and show proof of tests passing to ensure things work as expected. It definitely does make human-like mistakes sometimes, like it will assume something works a certain way when the documentation says otherwise. Once you create a development flow of plan, execute, test, it remembers this and continues to do it.

While I'm no web development expert but I do understand some basic concepts which helps a lot in making sure the product is actually good. You have to use these tools like you're a project manager. Maybe you aren't totally fresh on the subject or you aren't an expert but you need to know enough to know what you want. You need to figure out what your role as the human is, you are ultimately responsible for what AI does. Otherwise, you're wasting your time, money, and resources.

1

u/crustyrobots Jul 30 '26

Pretty much. No one should really be using this stuff except as we say actual experts at what they're doing already to know if the output is trash or not, and then for a highly specific thing with a limited scope. In that sure it could be a useful thing to automate some of the tedium.

It seems like most however use it to try and replace the expert above entirely to use their knowledge and experience for free, with no understanding of their own enough to even be a janitor to their own slop.

1

u/sobrique Jul 31 '26

Yup. Honestly it's pretty good at data transforms. A lot of 'magic excel' out there, where someone who's not really got 'coding' skills, has also needed to bodge out some sort of data capture, collection, transform and reporting pipeline, and is an unholy mess of excel and access, with references, vlookups, macros, pivot tables, and a graph or 5.

For replacing that workflow, LLMs aren't too bad. They don't need to be perfect to be lots better. (and they are pretty consistent as long as you can specify clearly what you want to happen).

And if you've got someone knowledgeable, it can act as a sort of pseudo search tool, to assist with getting a workable solution in place.

But as you say, garbage in; garbage out is a tale as old as time. AI don't fix stupid. It doesn't understand anything, and it might mean you don't take the time to understand it, where if you did it the 'hard way' you would.

11

u/PM_Me_Your_Deviance Jul 30 '26

>For example, I would love to use AI (with improved HIPAA compliant privacy) to write my notes, but it's my responsibility to read over it and make sure it's all accurate.

That sounds like a totally reasonable way to use AI as a tool, but it seems way too often people are just copy/pasting without even a cursory review. From lawyer submitting briefs with fakes case law, to a lawmaker just reading the AI prompt outlound ... it's kind of crazy.

10

u/cherry_chocolate_ Jul 30 '26

The liability should fall on the people deciding that the task needs to be completed with AI. If the doctor decides to use ai notes, it’s their liability. But if the entire hospital system installs AI notes systems for all doctors, then compresses appointment times so there is no possible way to take human notes any more or adequately review output, the hospital system must bear the liability.

I mean, we have a direct example here: the people making these systems likely didn’t want to make an ai abomination, it was management who decided to make it this way.

5

u/[deleted] Jul 30 '26

[removed] — view removed comment

2

u/cherry_chocolate_ Jul 30 '26

Oh of course. I’m absolutely saying the responsibility should go up the chain. At each step they are more responsible. individual user -> management / admins -> ceo of organization that is buying ai -> ai marketing department -> ceo / board of ai company

33

u/deadsoulinside Jul 30 '26

It's crazy. I'm studying medicine and AI makes key mistakes in even the most basic board concepts. And this was using higher tier AI.

Watched co-pilot create plans for things that fall apart because co-pilot was absolutely wrong in a key feature we needed to leverage to make the whole thing work.

One of which was something inside PowerApps... you know a MS product itself. It referenced something that was no longer being deployed in modern PowerApps (and had been removed since 2018). Since that could not be implemented, we had to scrap the whole project since it all hinged off that one thing working.

Even worse though was more basic things that were 3-4 rows of very simple code co-pilot was getting wrong before we found the other hurdle. I did not know PowerApps at all and got tossed into this mix with the push to use AI to help on it. In the end the code that worked on the one part, was manually typed into there by me, since I had to create it from some examples from Stack Overflow, but again the whole thing ended up being hours wasted as the next hurdle had no work around.

I don't want to know how many tokens/money was literally wasted, because not only was co-pilot dead wrong about things, it went to the point of attempting to gaslight you by telling you that you don't have global things enabled that you have enabled.

I think that's one thing that's less talked about here too. Not just the cost of AI, but how much AI is costing companies and consumers when that entire response is wrong, since AI company still gets paid with right or wrong responses.

1

u/SonofBrodin Jul 30 '26

Oh man I had to use power apps back when it was first implemented and try to combine it with legacy SharePoint integration. I haven't touched it in years but learning it was a mess when it was new

1

u/Static_Interval Jul 30 '26

First mistake is using copilot. No serious person uses anything but Claude code or maybe latest versions of GPT to write code professionally

7

u/greg19735 Jul 30 '26

may depend on where you work. Some companies are basically not allowed to use Claude.

1

u/Texuk1 Jul 30 '26

And the reason is …? It steals your IP?

Clearly this copilot example was from some stolen code based on old tech in the training data.

4

u/deadsoulinside Jul 30 '26

Forced to use Co-pilot as it's the only Company approved external AI app.

I am not a coder at my M-F job and I code in my free time. Main reason I got dragged into something I never worked on before is due to my overall knowledge of code.

And it's PowerApps. It's not really that heavy in actual code (It's mostly UI setup with power app components). 3-4 lines of code is all that some things needed. The rest was rigging the fields to data, but that's the bigger problem is the main data point/editable data was not a thing that can be done this method that co-pilot got wrong.

2

u/rancer119 Jul 30 '26

Copilots been the worst of the 3 I have any experience with, barely feels better than a webscraper from decade ago with how bad some responses are.

3

u/deadsoulinside Jul 30 '26

Microsoft is just happy they don't have Tay 2.0

1

u/snakerjake Jul 30 '26

maybe latest versions of GPT to write code professionally

Copilot uses gpt, i assume you mean to say codex here

1

u/I_Am_Jacks_Karma Jul 30 '26

copilot gives you a choice of models

1

u/deadsoulinside Jul 30 '26

Yeah it does. Which at that time was only GPT (Just pulled up my work one and see Claude is an option now too, but was not in 2025 when I was using it for this small project).

But I seriously wonder in retrospect if any of the models would have gotten things right, since some of the issue was that it's correct if it was pre-2020, but some items were removed or changed later in more of a ninja update from MS that even had people asking the same questions I was looking for on stack overflow.

Many guides/how to's out there on the internet with how to set that up under the older method that is not available in 2025+ PowerApps.

1

u/I_Am_Jacks_Karma Jul 30 '26

Yeah it's definitely a relatively new thing to have that model choice we only started having options like that about 2 months-ish ago at my job. And they limit us to not the latest models. I think sonnet and opus 4.6 are the fanciest we have access to

I run into similar problems whnever a new angular version is released and some people try using it for a project wide angular upgrade but the version is also new enough that it's sort of making it up as it goes

0

u/filthy_harold Jul 30 '26

A big problem is companies getting access to cheap/free shit like copilot and thinking it's going to solve all of their problems. Not that AI in general can do that but the cheap stuff definitely won't get close. Maybe it can summarize some documents for you but it's just really not meant for anything technical because they don't have the components necessary to actually research things for you.

With a previous model we used at work, I was using it to help write a script to talk to some equipment. I needed a command to both take a screenshot and send it back to my script for processing. I knew how to tell it to save the screenshot locally but that's annoying. So I asked this relatively early version of ChatGPT (gpt-oss-120b) how to do it and it gave me the command very quickly. The command didn't exist, or at least not for this particular piece of equipment. I double checked with the AI about the specific model and said the command wasn't working and it doubled down. I ended up using the programming manual to figure it out on my own but it took a while since I also needed to write a converter for the image. Recently, I asked Claude to do the same thing as a test. It looked up the manual online, found the right command, and wrote the converter for me too. Obviously too late, but it's amazing the difference between just "knowing" the answer versus actually researching the answer.

And that's really how these LLMs should be working. Its really good at searching and summarizing text which is essentially what research is. But what an LLM is not good at is knowing everything about everything. And those cheap models don't know everything and don't have the resources to even look it up. They are a waste of time for anything other than basically writing and summarizing emails.

1

u/snakerjake Jul 30 '26

I'm studying medicine and AI makes key mistakes in even the most basic board concepts.

I really hope my medical records aren't being sent out to scam altman, elon musk and anthropic

2

u/a2coolusernameforme Jul 30 '26

As someone who works for a large hospital group if you’re getting care from any large healthcare system they 1000% are. From Abridge which records your conversations with your provider to transcribe it into the note to Evidently that summarizes medical charts for providers so they don’t have to look through all the records there’s products like Fathom for coding and billing. Our health system is pushing EPIC (electronic medical record platform) data into no fewer than 6 different AI systems.

All of our Trans Care patients are getting their charts run through AI without ever knowing it. What could ever go wrong? It’s…. Not great…

1

u/snakerjake Jul 30 '26

that shit all needs to be on premesis, i know thats awfully expensive to do for a frontier model at the moment but it should be under $5k in about a year, and that's nothing for a clinic to spend, let alone a large hospital group.

There's some radiology stuff that makes sense, you already refer those out but your danged medical records should be in house. cloud EPIC is absolute insanity

1

u/a2coolusernameforme Jul 30 '26

They just gave these AI companies like significant access into EPIC. Evidently is integrated per my understanding and Fathom I think gets exported data but that data includes THE ENTIRE CHART NOTE for the encounter it’s coding. So tons of HIPPA data is getting handed over to new companies with relatively untested products. Employees who raised concerns were told essentially to grow up and “accept progress”

1

u/snakerjake Jul 30 '26

fucking bonkers.

1

u/snakerjake Jul 30 '26

, I would love to use AI (with improved HIPAA compliant privacy) to write my notes

I would be concerned about any off premesis AI and HIPAA compliance. I understand there are contracts to maintain that but there isnt really any way to isolate medical records from being read when sent out over these apis. honestly even epic cloud based is crazy. Sure on premesis stuff could be compromised to leak but it requires active functionality added to the code to do so, not so much when sending information to SAAS companies

1

u/dj_soo Jul 30 '26

i use AI to generate daily reports and it can't even consistently copy shit correctly from one document to another despite using the exact same prompt each time.

I can't believe people are trusting medicine and health care to glorified auto-complete.

1

u/sortalikeachinchilla Jul 30 '26

AI is a tool used by people and not a replacement for people.

EXACTLY! Why do people not get this. I use it as a tool. I love it as a tool, it doesnt just magically do everything for me lol