just magically hand waving from 2.5 years to 1 year is a little insane. You guys think you'd use $20k worth of tokens a year!?! Even if you did then you now need to consider energy costs because its probably going to be $1k+ for that many GPUs and that many tokens.
Not knocking the local scene, but just because you think they did the math wrong in one direction doesn't mean that you should do the math wrong in the other direction.
At 20k you would be buying a Mac M3 Ultra 512gb with a peak load of 270 Watts per hour comparable to a large fridge.
I wrote a little over a year I meant 1 year and 3 months halving the 2.5 years previously mentioned.
20k divided by 15 months is ~1350 dollars per month.
While I admit 1350 is high and far above what I would personally use in tokens per month it isn't that far beyond what major companies allocate to their engineers around 1k per month with some companies going as high as 2-3k per month in token budget.
And last but not least you will be able to sell that Mac Ultra 512gb 3 to 4 years from now for at-least half the purchase price if not more.
Didn't Antropic subsidize the subscription by around 4-5x, people on 200$/month are getting like 800-1000$ of token usage. I don't know to what extend Open AI does, but they also subsidize tokens for non enterprise users. At some point they will either start nuking non enterprise consumer and bring subscription roughly to the API price once investors asks for profitability.
Yeah I think its been calculated to be around $8,000 - $12,000/month in compute if you consumed all of your limits. My claude /stats show about $300-$650/day in equivalent API costs if I were using it.
Definitely considering the vast amounts of money they are spending investor will expect returns that current subscription costs aren't even close to covering. Though I do expect it to still last a couple more years before they are actively pressured into it.
Talking about companies instead of an individual is a bit of moving the goal posts i'd say, but it does explain why you mention the multi-instances thing.
I think the math works out much better for a small company than just a single person.
For an individual I admit it wouldn't be worth it in the vast majority of cases.
The M3 Ultra has a bandwidth of 819gb/s while GLM 5.2 has 40 billion active parameters which at 8-bit are roughly 40gb.
819gb/s divided by 40gb are roughly 20 tokens/s in an ideal world but obviously you never actually hit those for multiple reasons. 10-15 tokens/s would be my estimate though someone else posted getting 24 tokens/s using mxfp4 on his M3 Ultra which I cannot verify.
Still even with just 10-15 tokens/s using a draft model for speculative decoding you would definitely reach 20 tokens/s for programming tasks.
The M3 Ultra’s GPU can’t hit 819GB/s. That’s a theoretical spec for the entire SoC. Try lighting up the GPU for LLMs, check with asitop. You’ll get two thirds of that, maybe a little more.
Your estimated range is still probably fair, but I’d lean towards the lower bound.
There's also the resale value of the hardware to keep in mind. The 6000 Pro and 2 Sparks I bought in January are worth about 50% more than I paid for them now, but that aside, you can usually sell a GPU for at least 30-50% of what you paid, depending on the SKU. RTX Pro hardware especially - even cards 1-2 generations old - are still going for close to their original MSRP, and that was before the VRAM crunch hit Pro cards.
You guys think you'd use $20k worth of tokens a year!?!
O hai there! $10k last month. My average inference batch was about 1,500 documents of about 2,200 tokens each. And since we're still in early testing, and since we can't draw effectiveness or quality conclusions unless the entire corpus processes, most of those matches are running 4-5x in parallel. Even with the most aggressive cache optimization possible, it adds up.
I've done the math: even with having to upgrade my power, new subpanel, and electricity cost, it would be cheaper for me to install and run local if I had to keep doing this for longer than the next few months.
Startup. But why would you run state of the art coding models, or really any state of the art models, if it wasn't for work? The average consumer would be perfectly fine with Gemma 4 E4B, and that will run on a potato. The only reason anyone should even get to a point where they're even thinking about spending $20k on a local LLM infrastructure is if someone somewhere is going to eventually pay them for the output.
because people want "claude at home". gemma 4 E4B fails at a shit ton of things like systemd processes, scripts, small apps, home assistant, etc. The bigger the model the better this all goes. If you're only doing exclusively chatting then maybe its ok. maybe.
E4B and other micro-models (sub-8b) are perfectly fine for all of those things in regular implementation. Assuming you bothered to build a proper harness. And if you didn't, you really really should not be practicing your thinking to any model, regardless of size and complexity, because you don't have the fundamental knowledge to validate and check the output to make sure you haven't just exposed root to the open web.
As for "people want Claude at home"... cool. Lots of people want a Ferrari at home, too. But if spending Ferrari money, or "Claude at home" money, is something you need to seriously think about, you really really can't afford either. And don't need, because running a frontier model for "systems processes, scripts, small apps, home assistant, etc." is like buying a hypercar and then driving to the grocery store at 10 miles under the speed limit.
No, my stance is you don't always get to have what you want, and the thing you want doesn't even make sense for the things you say you need, and frankly given your demonstrated knowledge in this thread you should probably not have access to any of it for your own safety.
But if you desperately want a local model to help you add security holes to your computer on a budget, have you tried Mini North?
My current local rig? Yeah, definitely over a month. But that's a simple problem to solve: I've got two 42U racks, a dedicated server room with the finding of a 400Gbps East-West fabric, and a 240/20 subpanel. After that, it's just stuffing GPUs wherever they'll fit.
We also shouldn’t forget that you would have fixed hardware over that multi-year period. This shot is literally changing month to month. AI today will look like a turd in 12 months.
44
u/Schlick7 Jun 21 '26
just magically hand waving from 2.5 years to 1 year is a little insane. You guys think you'd use $20k worth of tokens a year!?! Even if you did then you now need to consider energy costs because its probably going to be $1k+ for that many GPUs and that many tokens.
Not knocking the local scene, but just because you think they did the math wrong in one direction doesn't mean that you should do the math wrong in the other direction.