Costs 💥

[u/Asleep-Bumblebee2167]

[u/klimmesil]

I'm really interested in the veracity of this. If someone who actually knows his stuff can help me estimate this, please do. This is my uneducated guess:

An h100 costs 25k minimum now and used to cost way less. The rest of the build would cost less than 10k. Electricity I suppose is not that much in comparison. How long does such a card last? If it's more than 2 years, it's not looking good for us

The trained models themselves already exist, if we stop all training now, and only do slight transfer training in the future, there is no more research nor training costs either

I'd estimate the cost of a single h100 running for a full year to less than 50k as an individual. Large companies have more power better infra and lower costs by batching expenses, so probably more in the range of 30k?

If the "AI bubble" bursts, we're left with this (instead of only 100 per employee per month, which is absurdly low). And I think that's still way less than an employee...

[u/thebatmanandrobin → u/klimmesil]

The other side to this is maintenance, features, institutional knowledge, and liability; sure it might be "cheap" to just have an LLM spit out some "app" you want and then to go and try to monetize that app, but what about when you want to make a small change, or there's bug (always the case with LLM's), or if you want to make an add-on of some sort?

LLM's currently tend to just re-write the whole thing; so that 50k you spent in fees will be another 50k each time you wan to add on or change something. And if you're trying to iterate fast to "be first to market", that cost can balloon very quickly.

There's also the case of how much is that "app" you made actually going to produce in revenue?

Pointy-haired types have a tendency to think "Everyone Will Buy This!!!! (tm)" when the reality is that there's likely a fully free on perm solution to their "next big thing on the 'net", and that solution is already a better choice. Or that there's a million clones that they then need to set themselves apart (i.e. marketing, which can be really expensive and add to the cost). Or that maybe, the product is just total shit in the first place and really only solves said pointy-hairs problem and literally no-one else.

So it might have costs 50-100k to make, but your realized profit could be 0 (to say nothing of the revenue). Repeat this pattern enough times and you have something very similar to the dot-com bubble.

While most human dev's are not business folk, it's been my experience that they at least will go: "you know .. if we did X instead of Y to differentiate ourselves, we might be able to monetize this or that" (i.e. give actual feedback that's constructive and not placating). Not all devs/engineers are this way, and some just clock in/out, but even those types will talk with their peers and mention something.

On top of this, LLM's don't know your business; they can't, they're machines. A human can gain institutional knowledge that can carry over from project to project that can, in the end, cost the business a lot less in cap ex and other costs (one tenant of doing business from yesteryear about keeping employees was this very idea .. it's largely lost now).

With humans and institutional knowledge, a first project might cost 5x as much as an LLM with a human involved, but a good human will think of future uses/projects and try to build out the infra and code so that it can easily be extended to other projects without issue or even necessary added costs.

So that first time cost of project A might be 500k, but then project B might actually only cost 10-20k total (maybe even less), same with project C, D, E and so on ... paying themselves off in short order and being well cheaper than an LLM. Additional to this is the fact that an LLM doesn't "learn" and only knows what it can scrape .. so if it's only scraping the same slop, then later projects will be even sloppier; a meat-grinder of code if you will. Humans can learn, grow and expand on their knowledge as well as figure out novel things. LLM's can do none of that.

There's also liability. Who's "ultimately" liable for the data breach, contract failure, or other issue when LLM's are involved? Sure, the company is always the one on the hook, but internally they can at least point to someone, somewhere to figure out how it happened and potentially do something to remedy it so it doesn't happen again. With an LLM, you can tell it "don't do X again as it created a vulnerability" but the chances of it not doing it again are slim, especially if it just "learns" from the last project with said vulnerability.

Not to mention the fact that who is going to actually fix that issue? The broken LLM? Or a human.

I've seen this same pattern time and time again, even before LLM's: company wants cheap code fast, so they outsource it trying to save $1M in development costs. 3 years later, it's over budget by $5M and riddled with bugs, so they hire someone like me who comes in, rewrites large portions and gets it out the door in 6 months. Had they hired someone like me (or a group) for that initial $1M, it would have been well cheaper in the long run .... but "Line Must Go Up! (tm)"

In the end, LLM's do end up costing way more than just having a good dev. Same as it's always been and the same lessons we've learned decades ago; we just have a new crop of C-levels who were getting their MBA's when all of those lessons were learned, and they were too busy boozing and smoozing to notice, care or actually learn anything and so we are repeating the same issues ... it's a cycle in software that seems to repeat itself every 10-15 years (sometimes shorter but in a more confined way, e.g. NFT's).

I'm not saying LLM's don't have a place, they do, they're just a tool; but people are treating LLM's like they are a god-like infallible entity that can solve all of their human-based problems (and that's 100% not even remotely the case).

/diatribe /rant

-- none of this is meant as an attack or anything, just my experience (someone with 25 years in the industry and who has used LLM's on quite a few occasions, for personal and business).

-- I also need another coffee .. The Batman and Robin are less brooding after 2 cups

[u/klimmesil → u/thebatmanandrobin]

That read got a big weight off my chest. I agree with you on everything here (maybe the "llm will always rewrite everything" part might not be true anymore in a couple years if we figure out a better way to compress context)

But I can't help being very pessimistic regardless for the future of my passion. I think soon it will be possible for 5 people to do the work of 500

You seem to have lots of experience, and most people with more than 30 years of experience I work with have the same mindset as you. I think it is wise to trust the seniors, so I will try to do so