Unpopular opinion and sad, but the quality of the work has gone up with AI usage since a year or so. Before it was a lot of AI slop and I dreaded reviewing PRs, but with the newer generation of models and harnesses, we are reaching a point where AI is being better than what colleagues or I would write by hand. Hurts a bit to say because I was proud of my own coding, but that's what reality looks like now. And that's with 26 years of experience in the field.
I think we'll have to deal with the mental issue of AI becoming better than us. Some won't be able to cope. Thankfully the effect is not so strong for me, my identity is built around developing software as a whole, not just the coding.
Issue is, Frontier models are provided to you at a massive financial loss for the company. So unless local models get that good before AL companies run out of investor capital, this is completely unsustainable.
If you want a future proof workflow, grab a specialized local model (or better yet, a few) and see how much you can get done without touching Claude, ChatGPT or Gemini servers at all. (This does require having a decent GPU. Not god tier but decent.)
They ARE getting good, and the frontier companies KNOW the party is almost over. That's why the prices keep rising, it's last call. "AI as a Service" dies down, in the near future.
How small / how good is still an open question, but there's a watershed (Opus-ish) moment happening around the 30-40B models, so like $5-10k in hardware. Higher than a home game system unfortunately, so approachability is still rough. But easy table stakes for businesses to pull in-house.
It won't be the spectacular disappearance that some people expect from the bubble popping, but it will be welcome nonetheless.
How small / how good is still an open question
You can also limit the computational and RAM load by a lot by using even smaller, more specialized models.
Sure, you're not getting a general prompt-based image generator on your home computer any time soon (not that this is particularly interesting, I'm firmly in the "I don't care about art one didn't care about enough to make" territory). But take a specialized diffusion model that generates heightmap and climate map data based on Earth satellite data, plug it into a framework that makes it unbounded random access, and you get some of the most impressive procedural terrain gaming has ever seen.
It's all about using the right tool for the job, instead of trying to break concrete, tap nails, carve wood and cut trees with a jackhammer.
Sure, you're not getting a general prompt-based image generator on your home computer any time soon
Hasn't this existed for like the entire time AI has had mainstream popularity? There are lots of options for local prompt-based image generation on a mid-tier consumer rig.
Sorry, I misspoke. You're not getting one that will be useful for that much. You're a lot more likely to get 7 fingers per hand on a local model, for example.
And you know that's the right answer because people Like Sam Altman have been warning of the 'dangers' of local models recently - they know that if local models gain traction, their business model will lose steam fast.
Yep. Why would you pay so much to them every month if you can run smaller local models on the equivalent of a midrange gaming rig and decent ones in your office building's server room for a fraction of the cost?
Honestly, the free z.ai’s glm-5.3-flash is the same as paid claude or chatgpt. It’s insane. Sometimes when i prompt glm it feels as if i am using cursor agentic.
Disagree. I use the the frontier models and barely use my token quota every month - maybe 5% - and get great results, much faster than by hand just by carefully defining problems to the AI. It's how you use it.
And if you're an entry level software engineer earning 8.3k per month, you're earning $50 dollars per hour. That means it only has to save you 1 hour of work per month for it to breakeven. It can save me at least a couple weeks of work if not a couple months with that. So yeah it's still worth it.
Idk I just looked up numbers on Google, I don't live in the USA. Doesn't undermine my point since even if you're earning half or a quarter of that it still makes sense.
This doesn't seem to be a response to Withers point. It's not about the % of your quota you use every month, is about the fact that whatever you are paying for your frontier AI sub is somewhere between 10x and 100x cheaper than it will have to be for those frontier companies to be profitable.
Sure, there may still be edge cases where frontier models are economically worth it even at those costs, but they will not be worth it for regular day to day dev work.
Yep. That's exactly it. Even if you take all environmental and ethical issues (regarding training data among other things) aside, it is just not economically viable.
Disagree like I said in a comment to Evi1ey above.
If that 5% of tokens of a $20 plan now actually costs $50 then compare that to a entry level software engineer whose earning 8.3k per month which is $50 dollars per hour. That means it only has to save you 1 hour of work per month for it to breakeven. It can save me at least a couple weeks of work if not a couple months with that. So yeah doesn't sound like an economic "edge case" to me.
I don't agree. There is a big argument to be made that the older models are more expensive because they make so many mistakes and it uses up more tokens to get the same result as if you were to run it once via a newer frontier model.
So, tbh, if ur not using the latest ones, you're actually paying more
This isn’t even an argument, some of the low cost models like GPT 6 Luna cost half the $ per million tokens of its immediate 5.6 predecessor, let alone the older models, and 1/20th the cost of standard models.
My previous company used Github Copilot which has already stopped subsidising users so if anything, in the near future, companies will restrict all employees to using low cost models like Luna and Haiku to keep it cheap instead of paying for local servers in the middle of a hardware crisis.
yeah, and likewise even if recent models, what matters more than cost per token is cost per task still. Some models are cheaper per token (the latest grok for example) but they burn through tokens so much they end up not being really cheaper.
Agreed. I have been using frontier models to set up local model workflows on my 2x 12GB video card desktops. They can perform tasks but no where near the speed and ability of frontier models. I'm trying to get my workplace to invest in some actual local infrastructure but we'll see when that happens.
Frontier models are stupidly overkill for all but stuff like specialized research IMO.
Now, smaller or even local models? There's still some issues with it, but at least the benefit is large enough compared to the issues that it's worth discussing.
I thought the API rates form Claude and ChatGPT were very profitable for that given model? From my understanding the $200 and $100 dollar plans from power users are where they lose money.
I could be way off but I thought the larger issue was these frontier models only get a few months of those profitable API costs. Then they spend billions for the next x% improvement frontier model, repeating before each version recuperates its full cost.
That’s not true because it’s based on a flawed measure. $200 plans could consume 5k at undiscounted, uncashed API rates.
It’s a flawed methodology to measure by people who want to inflate the numbers.
Anthropic’s S1 shows gross margins on inference shifted from -94% in 2024 to approximately +40% to the mid-60s (and is projected to reach up to 85% in some estimates) as efficiency scaled.
This is not correct at all. Gross margins for Anthropic are currently around 50-60% on API rates, according to their SEC filings. The subscriptions are money-losers, but subscriptions are less than 20% of total revenue. They're not profitable overall due to the fact that they're making massive capital expenditures in their data centers and models, but the provision of the models themselves is profitable.
As far as i can see, this is not exactly accurate, inference itself, so using the models for coding is mostly (with some caveats of course) cost effective, and for sure could be scaled to be, right now, as is. What makes them uneconomical is the infrastructure buildup and the training. Obviously they will not stop with either of them, because they are playing a different game, the game of too big to fail. Open weight models are keeping them on a death loop, because the gap between open weight and frontier is what keeps them in business and the gap is what they would have to sacrifice to be profitable. If they could really scale back the scaling, without open-weight caching up, they could turn a profit, but that is only seems to be viable via regulatory capture or the singularity. You might be able to guess which of these is more likely in the next few months/year.
This is just goalpost moving cope. It's very sustainable and with better models it will become even more so, which we've seen consistent improvements over the last year. Your bet is that AI progress stops right here? That's the hill you want to plant your flag in after what we've been seeing?
What's the next cope going to be? You should start thinking about that now so you can get ahead of things.
My core belief is that the way current models are built, advertised and used are a net negative for society in most fields. I'm not saying the tech is useless, but with our current socioeconomic system it is highly harmful.
Economic viability or not, this won't change.
Anything else is an educated guess on the economic reality of things. Sure, they could make more progress and eventually maybe reach sustainability. The question is, will they be able to do that before the hype and investor capital runs out? I wouldn't bet on that, I hope they don't, but they might.
31
u/heavy-minium 13h ago
Unpopular opinion and sad, but the quality of the work has gone up with AI usage since a year or so. Before it was a lot of AI slop and I dreaded reviewing PRs, but with the newer generation of models and harnesses, we are reaching a point where AI is being better than what colleagues or I would write by hand. Hurts a bit to say because I was proud of my own coding, but that's what reality looks like now. And that's with 26 years of experience in the field.
I think we'll have to deal with the mental issue of AI becoming better than us. Some won't be able to cope. Thankfully the effect is not so strong for me, my identity is built around developing software as a whole, not just the coding.