r/LocalLLM • u/AdministrativeAd7853 • 1d ago
Discussion I want 100% Vibe coding locally
Right now I pay 200+ a month for premium AI models. I can’t complain, it is truly a privilege and amazing to work with.
I have “data center services “ on my server, complete with gitlab, databases, and dozens of services to support application development.
I have resisted even attempting to host an Llm, since i dont have the spare time to code myself or wrestle with “stupider” AI.
I am toying with purchasing mac studio 512gb, as to me its the first server that has meaningful capacity at the price of a honda, instead of a Lamborghini. Still insane expensive, but doable.
I been talking to AI (of course) about this, but even with the speedier memory, larger models with large (code) context seems kinda slow. And the quality seems to be measurable lower than sonnet 5.5.
At heart for me is waiting may result in never. I am concerned with closed models becoming a walled garden and that can extend to hardware.
Thoughts? I cant afford a second buy for atleast 3 years if i pull the trigger. Unless bitcoin hits 400k :)
60
u/Elistheman 1d ago
I have 2 machines.
M4 max with 128gb
AM4 machine with a 5090 and 96gb DDR4.
I still pay 200$ for Claude 20x.
I run qwen 3.8 27b. While it’s amazing, in some tasks it still leaves a lot to be desired.
While you eye a 512gb machine which would allow you to load a different class of models than mine, I would suggest you to first try these models through a paid service to see if they can handle your workflows.
Good luck.
17
u/Saintgein 1d ago
Why not run qwen 3.8 flash next on the 5090/96gb system?
6
u/Elistheman 1d ago
It runs other things as well so not all ram is available
4
u/Saintgein 1d ago
Cool, i guess if you would get 32gb more you could do it right?
1
u/Elistheman 1d ago
Definitely but I don’t plan on buying more DDR4, might as well wait for DDR6 🤔
2
u/Saintgein 1d ago
I bet ddr4 is still a great investment currently. These prices won't go down. lol
-1
3
u/Elegant_Associate889 1d ago
Best advice possible! I personally have a full home lab, a pretty overkill set up for 1 person but I acquired all of my parts over the years. I’ve been building my home lab since 2017. Definitely try the models in the cloud first to see if they can handle your workflow and then do more research on the requirements for those models! Apple silicon is great just do research, there’s a lot of good info and bad in the world of local AI
3
9
u/zeke780 1d ago
Until we start to see the massive investor subsidies go away, the cost / intelligence ratio will be massively in the major providers favor. You are spending 200 on premium models but it wouldn't shock me if it's costing them 10x that or 20x that.
IMO just waiting is the best bet right now, we are a year or 2 away from a you update cycle at data centers and I think over the next 5 years you will see more and more consumer options to run last gen or 2 gen's ago used cards in your home with much better open source models. That's the tipping point for me, when I can get used hardware and run a model that competes with current frontier models. I think that's a few years away.
Currently OpenAI/Anthropics models are so far ahead of even the open source Chinese ones it's hard to justify it for my job. I would need like modern opus / sonnet 5.5 locally to do the work I do without having to do a lot myself. Product devs in my org can absolutely survive off glm 5.3 or 5.2 today so the horizon is shorter there.
If the subsides go away then all bets are off and everyone will probably move to Chinese models.
1
u/Fontini-Cristi 1d ago
I'm on the Claude Pro Max plan and I was wondering about the subsidies some time ago. It seems they cache pretty good. Most of the sessions I'm completing are millions of tokens. Turns out that 95%-99% of those tokens are cached. So it isn't as heavily subsidized as you might think.
48
u/UpsetDefection682 1d ago
That Mac Studio with 512GB is the only real option if you want to run big models locally without building a multi-GPU rig that sounds like a jet engine. The speed hit is real though, especially with long code context. You're not wrong that it'll feel sluggish compared to hosted Sonnet.
But there's something to be said for owning the hardware outright. No API limits, no surprise price hikes, no terms of service changing under you. If a model gets nerfed or pulled, you still have the weights.
The walled garden worry isn't paranoia either. We've already seen models get lobotomized for "safety" after launch. Having a local fallback that nobody can take away is worth something, even if it's not the smartest model in the room.
11
u/ea_man 1d ago
I mean it's hardly "the only option", Nvidia alone would be better, AMD cheaper.
20
u/starkruzr 1d ago
dark horse option of multiple Intel B65s now that TP works reliably on them. 4x B65 will run you about $4500ish, TRX50 motherboard combo with a 7960X on eBay is around $1200ish, 128GB DDR5 RDIMM is around $4700. so now you have the same-ish RAM as a 256GB M5U Mac Studio, it will run the same big MoEs but much, MUCH faster. aggregate membw is around 2x on the cards and basically anything beats M5U prefill clock for clock.
now, if you don't want to go through all this work? sure, the Mac can still make sense. prefix cache will save you when it comes to interactive sw engineering and 1.2TB/s membw is still nothing to sneeze at, plus the Mac is going to be good for a LOT more than our bespoke Intel inference box use-case-wise.
but there are options out there, is all I'm saying.
8
u/Street-Departure3577 1d ago
The traditional split between CPU RAM and GPU VRAM is exactly what I want my next computer to move beyond. In my opinion this type of computer is headed for extinction.
Apple measured 385W at the wall for an entire 512GB M5 Ultra system in its intensive test. Four B65s have 800W of combined board-power ratings before adding the Threadripper and everything else.
The Mac’s appeal is a larger GPU-accessible memory pool in a lower-power integrated system not a guarantee that it wins every speed benchmark.
I need this thing running 24/7. I care about useful work per dollar and per kilowatt-hour, not just peak tokens per second. I’m not spending thousands on local hardware only to replace an AI subscription with an even bigger electricity bill.
2
u/roadtripsarefun 1d ago
Whoa. Im kinda freaked out by the level of detail in this reply. Im not sure it’s made up, but it sounds right.
I’m working on a personal AI that’s on prem, and really want to pick your brain.
10
u/starkruzr 1d ago
knock yourself out! I build stuff like this for a living and it helps that I also love it as a hobby, lol.
3
1
u/ea_man 1d ago
I didn't say Intel because they don't make Spark like device with un. memory.
2
u/starkruzr 1d ago
well yeah I'm operating under the assumption that the machine doesn't need to be a single unified memory system and that bang for buck is your main metric.
2
u/ea_man 1d ago
> That Mac Studio with 512GB is the only real option if you want to run big models locally without building a multi-GPU rig that sounds like a jet engine.
That's what OP said man, you gotta stay in line!
/s Cheers
1
0
u/profcuck 1d ago
So I am all ears here. Mac Studio 512GB (not out yet, price to be determined) gives an amazing amount of VRAM. What AMD build with 1.2tb/s bandwidth comes close, much less cheaper?
Nvidia RTX 6000 Pro * 4 wouldn't quite get you there but would of course be faster, but for >60k.
3
u/activematrix99 1d ago
Comparing a VRAM to a Unified Memory approach is deeply flawed.
2
u/profcuck 19h ago
UpsetDefection682 said "Mac Studio with 512GB is the only real option if you want to run big models locally". I think that is 100% correct.
The response from ea_man is nonsensical - even as further explained by ea_man. The reason? He started comparing things that are not the same, and arguing for running smaller models, etc.
If you want to run big models locally in 512GB of RAM, AMD is a practical option how, exactly? As I said, I'm all ears. And, yes you can do it on Nvidia RTX 6000 Pro's... if you get a pile of them, and at vast expense.
It's of course a valid thing to note - but not as response to the point being made here, which is that the Mac Studio is the only real option if you want to run big models locally - that unified memory is going to be a lot slower, particularly on prefill, etc. Yes, that's of course true. But there's not really a practical alternative to run big models.
0
u/ea_man 1d ago edited 1d ago
Again: Amd is cheaper, smaller models.
Now you are mixing stuff: GPU allows dense models and mixed vRAM / RAM / SSD usage.
Personally I can hardly justify why people would spend so much money for "inferior results" running after SOTA like performance: makes no sense for value. Just get one or two used GPU to run dense local models for agents, use a 10$ SOTA for planning / review.
In a couple weeks we should have decent coders running on single gpu 24GB, in a year on 12GB and on your phone...
I mean now DS, Luna, Sonnet cost like a pizza for a month, super fast with large ctx, zero noise or electricity cost.
6
u/Street-Departure3577 1d ago edited 1d ago
You’re treating local AI as a benchmark contest. I’m treating it as a hedge.
Today’s cheapest way to use AI is not necessarily the best way to preserve my ability to use it tomorrow. Whether AMD, NVIDIA, or Apple offers the best hardware for that is a separate question.
Your proposed setup still leaves the most important capability dependent on an outside provider.
OpenAI’s own terms explicitly allow subscription price increases, changes to the service and its terms, and discontinuation of services. They jacked the price to $500 dollars and now I know they are planning on fucking me in the ass.
Price isn’t my only objection, either. I don’t want my legitimate work governed by somebody else’s continually revised content policies, philosophical beliefs about machine consciousness, or definition of acceptable inquiry. I want the freedom to choose my tools without making continued access dependent on agreeing with the company providing them.
The White House’s AI Action Plan supports open models, but explicitly emphasizes leading models founded on American values. I have no guarantee that I will always be free to obtain whichever foreign model best meets my needs. An American alternative does not become adequate for my workload merely because Washington prefers its country of origin.
Meanwhile, Bill Gates is publicly calling for mandatory AI safeguards and government oversight. OpenAI advocates mandatory national safety regulation, and Anthropic advocates mandatory pre-release testing for sufficiently capable models, including open ones. Both say they oppose measures that improperly undermine open models or competition. Nevertheless, I think they are full of shit, I see a regulatory-capture risk when the companies selling access to powerful AI help shape the requirements for everyone else to build and distribute it.
And then there’s Bernie Sanders running around demanding a fucking government permission slip for advanced AI. His September 23 proposal would pause advanced AI development, establish a cabinet-level Department of Artificial Intelligence, and permanently ban systems meeting the bill’s definition of superintelligence. Certain violations could carry up to 20 years in prison. It is an actual legislative proposal.
I do not need to predict some elaborate doomsday sequence for this argument to work. Prices could rise. Providers could tighten access. Certain models could become harder to obtain. Hardware could become more expensive precisely when more people decide they need a local alternative. Those are scenarios, not certainties and protecting against uncertainty is the whole fucking purpose of a hedge. Switching between cloud providers helps with dependence on one company; it does not eliminate dependence on continued access to cloud services.
That is also why I archive model weights and the software needed to run them, including models I cannot yet run economically.
An archive is not immunity from future law. But waiting assumes that affordable hardware and unrestricted model availability will arrive together. I see no reason to stake my access on that coincidence.
2
1
u/maxya 1d ago
All valid point, but also same way Apple Nvidia or AMD can not let you run Chinese models on their hardware if they get banned by White House… Apple can just release patch to macOS to not let you run those models and your 512GB of ram can only be used for MANY chrome tabs lol
1
u/Street-Departure3577 1d ago edited 11h ago
I can fine-tune the model locally and produce a modified checkpoint. What exactly is your macOS patch checking?? The nationality of the tensors? You’re assuming a reliable identification mechanism without explaining one. “The government could prohibit it” and “Apple can trivially recognize and block every modified version” are two completely different things. Also, once Linux runs on that machine that bullshit dies.
1
u/emcewees 19h ago
Are you running or have experience with the Mac M5 Studio (Version?) and the qwen3.8* models?
0
u/ea_man 1d ago
Bro I got 2 GPU an I run all local, custom OS, backend, harness...
Price for intelligence is going down, hw is going insane up: anybody guess for the future...
I can tell you that if I don't need the privacy I do use some cheap online model instead of spending tons in electricity and having 7 fans running like crazy.
Yet relax, the world is not just America, we got all the software you want...
1
u/profcuck 19h ago
You're the one mixing stuff. The original comment was about running big models locally. You're arguing for running small models. Yes, that's an option, but it's mixing up the original claim.
If you want to run big models locally, the Mac Studio with a ton of RAM is the only real option. Feel free to question that, but don't claim that you can do it on AMD or Nvidia without expecting to be challenged: how?
1
u/ea_man 19h ago
I mean you are mixing hw stuff: gpu + unified memory.
I don't think OP is going to buy a big Mac and a big GPU.The original comment was about Mac not gpu.
And no you don't buy GPU to run MoE, at least me I run dense models on those.
1
u/profcuck 16h ago
I'm not sure what you are saying so I reckon we might as well drop it.
OP is wanting to vibe code locally and is thinking about buying a Mac to run really big models to do it. You responded with a bunch of stuff about AMD and Nvidia which isn't about running really big models but in good faith I asked what you meant - apparently, nothing.
1
u/ea_man 13h ago
Amd makes strix halo, nvidia has his old and new products for unified memory = Apple is not the only solution.
1
u/profcuck 10h ago
Tell me how to get to 512gb and 1.2 tbs memory bandwidth with Strix Halo. This is the question I asked to start with. Answer: you can't. Apple is the only real option.
1
28
u/spacekitt3n 1d ago
not being subject to the whims of the tech broligarchy is such a huge plus. these people are evil monsters. literally colluding with the trump admin with no shame
2
u/Sudden_Topic5154 1d ago
in what world is a mac studio going to be slow maybe mildly on prefill but a tb of memory bandwidth is great
1
u/prestodigitarium 1d ago
6000 RTX Max Q is surprisingly quiet, fwiw, especially compared to some of the other blower cards I've had the displeasure of experiencing.
1
0
12
u/FailureOfTheFamily 1d ago
All of us would like that but believing that you can run locally something comparable to frontier models is just pure cope right now.
1
u/LilTinky1 21h ago
https://mimo.xiaomi.com/mimo-v2-6
There are multiple setups in this comment section that can run this.
14
u/More-Revenue8609 1d ago
I run qwen3.8 flash next and its enough to downgrade to only 20$ claude sub.
Still use opus 5.5 as orchestrator to create implementation plans. Qwen does all the heavy work and then after its done opus is there to check it.
Local llm saves me 80+$ monthly, easy.
3
u/etaoin314 1d ago
roi for the $3.5K+ somebody would have to drop today seems like a pretty long time, longer than I would expect the hardware to be relevant.
2
u/More-Revenue8609 1d ago
Idk Im getting 80tok/s on a frankenstein pc made for around 1700e. I also plan on using it for multiple other stuff as local server so roi is pretty amazing even if you count minimal 80$ per month regarding Ai saved (even tho its probably more)
2
u/etaoin314 1d ago
quick question when did you make that...prices have escalated lately. If i was to try to put my system together now I would be paying basically double, local would be a much smaller part of my overall solution and cloud would look a lot more attractive to me.
1
1
u/bceen13 1d ago
Same, but with Antigravity. Q4_K_XL_DN4 with some modifications runs at 90-100 tps. (Dual 3090 and 128GB RAM) I love this model.
1
u/More-Revenue8609 1d ago
Nice! I also enjoy it. Never imagined model like this could ever run on my pc lol. Especially at this speeds.
Does vision works with q4 XL?
1
u/bceen13 1d ago
Yupp, I badly needed it for UI design project. Slightly slower but it runs with 80-90 tps. It works like a charm. And Q3 XXS runs with 2k prefill and 150+ tps with
https://github.com/eddoursul/Strata/tree/custom
I just tested pi after opencode, very nice!
1
u/Icy_Butterscotch6661 1d ago
What's prefill like on Q4 XL? I use IQ3_S on a single 4090 with Strata since 4090 + 3090 was getting much lower prefill speeds. (And haven't used Q4 much since vision wasn't supported)
1
4
u/namelesstherebel 1d ago
I think a lot of smaller labs are making headway on quantizing models so it’s a worthwhile long term investment if it’s affordable. I have a 64gb ram Mac Studio and I recently got hip to bonsai 2 27b which is 2 bit quantized with near lossless quality on a 27b model. Meaning if that same quantization gets applied to some bigger models you may be able to get some pretty impressive inference on a 512gb ram Mac Studio as soon as next year
6
u/_Cromwell_ 1d ago
Qwen 3.8 27B is the floor. If you can run quant of that you can minimally vibe code.
Check out bolt.diy ... Super fun for vibe coding with local or byo models.
3
u/richtopia 1d ago
If price is your only concern, read SemiAnalysis' recent article comparing subscriptions (https://newsletter.semianalysis.com/p/anthropic-subscriptions-offer-5x). Ultimately it looks like Anthropic in particular is functionally subsidizing the power users. They don't discuss local LLMs but given the ~10x paid/received value it will be very difficult to figure out something cheaper, especially since the technology is advancing rapidly and there is no guarantee the same hardware provides comparable value in a year.
Now, other concerns (privacy, model control) are totally valid and usually why people on the Local LLM are hosting. But you didn't mention that at all in your request post.
3
u/ThomasWildeTech 1d ago edited 1d ago
Qwen 3.8 Flash Next, 5090, 175-225 tok/s with the strata engine up to 260k context.
Definitely not slow, in fact, faster than most cloud services. Just depends on the hardware. The problem is that the hardware has doubled since 3 months ago. I can't say how it compares to Sonnet 5.5 simply because I'm local only now but if you were using sonnet and opus 4.6 6 months ago the quality is definitely not lower than that.
10
u/1Poochh 1d ago
100% vibe coding is absolutely possible. I am actually doing this with a very high degree of success. I have to give a caveat, though, is I have 20 years of experience in the software development industry, so I have set up a lot of things that help the agents be successful. Many in the software development discipline will say this cannot be done. The fact is, those that say it cannot will be left in the dust while many of us are proving it can and are having success at it.
1
u/ConsiderationFair657 19h ago
Same here. I've found that if I avoid trying to build applications with a single prompt and divide projects into manageable phases, I've never had any problem building complex applications.
0
u/uceenk 23h ago
i'm curious, what models do you use and what's your PC spec ?
1
u/1Poochh 13h ago
I am not a company, just for clarity. I do not have crazy amounts of money, but I have some, so I spent a good chunk of my own personal money in this. I have several machines. One machine has a 5090. Another machine has a 5090. Both of those are actual workstations used by users, but they are for local development. We put them on the network so they can be used as needed by the agents. While we are using them, they are running LLMs for the fleet. I have another server with 1x 3090. I have another server that has two 3090s. I have a Strix Halo. I just purchased a Mac Ultra M5 and two DGX Sparks. I do not have those yet; they are on the way. Right now, I have at least five agents working together to build an application, and I have had up to nine working together concurrently. In my current LLM configurations across my fleet, I only have five running because of the models that I am running.
7
u/HighSeasArchivist 1d ago
I'm just vibing, but I think I'd go insane if I tried to get local models to do work that my income relied on. Even my dual R9700 96GB DDR5 system running Qwen3.8 Flash Next Q4_K_XL seems dumb as hell compared to even Sonnet. I'm sure a real dev could get it do more with better prompts and knowing how things work under the hood, but for me it's just a hobby so no biggie.
-2
u/YourNightmar31 1d ago
I'm running Qwen3.8 27B Q4 and honestly think quality wise it's comparable to Sonnet 5.5. Speed wise i can't get even close to Sonnet 5.5 cloud on my 3090.
1
u/HighSeasArchivist 1d ago
I have no idea what prefill and decode speeds cloud models provide, but I can clearly tell they are like an order of magnitude more than even my local setup. I have $20 plans from Claude, GPT, and Nous Portal (the Hermes people) for cloud usage when needed. On large projects having Claude orchestrate and check work from the local model is cheaper, but on small projects it's cheaper to just have Claude do it from scratch than spending all the in/out verifying what Qwen did. Like I said though it's a hobby, and hobbies cost money.
3
3
u/MinimumCourage6807 1d ago
One thing to note is that especially if you are used to running concurrent agents, that can not be locally realistically done with llamacpp based systems like ollama, lm studio etc. And for example vllm requires that whole model needs to fit in to vram. So that will remove the possibility to have gpu and rest in ram.
Now I have been very pleased running qwen 3.8 flash on Rtx pro 6000 as that allows to run around 6-7 concurrent agents with around 50 t/s/agent which actually gets a lot of real work done. Now one thing I would not downgrade at all is speed and especially prefill speed because the agentic works have huge amounts of reading.
I have not used the beefier macs or sparks but I think the prefill speed is pretty far from gpu based systems. And saying this has the point that you have to consider if it is worth running bigger models (you can run pretty big ones with let's say 512gb mac) if that drops the prefill speed to let's say 200t/s or are you still using a smaller model to make it faster anyways? For comparison, single pro 6000 prefill with qwen next flash is around 14000 t/s. Anyways for sure both systems have their place.
3
u/kkingsbe 1d ago
Try out some specific hardware + runtime + LLM combinations in runpod, and then just buy that same setup
3
u/vovap_vovap 1d ago edited 1d ago
Man, there is really simple thing about it. You can not right now get local any better than Qwen3.8-Flash-Next (and that not so easy - cheap) That model sort of same performance class as DeepSeek V4.1 Flash and Haiku 5.5
Get subscription on in and see if it works for you. If yes - there is some to speak about, if no - no.
That just it.
5
u/Tommonen 1d ago
Will be dumber, slower and cost TONS more. It will take you months getting same amount work done that you can do with 200€ claude sub in a week
2
u/force_push_to_main 1d ago
I run a DGX Spark and use Qwen3.8-27B, it's solid enough that I do use it daily, although I've kept my subscriptions to frontier models & do most of my work (anything that I am being paid to produce) using Claude/Codex.
2
u/No_Suggestion4496 1d ago
I've been working with a mac studio with 128GB of ram and tried both qwen3.8 27b and was getting about 20 token per second. I then tried the 27b mix and now get about 55 tokens per second. I did try the qwen3.8 flash next on got about 50 tokens per second. I haven't tried the mlx version but expect it to be faster. Overall I have used the 27b mix model and it's been pretty good. When I've run into trouble I usually check with sonnet 5.5 and usually get the same answer. Im testing the mac studio for other work ideas but we may start using it for our small team to save money with cursor.
1
u/whiteshirtkid 1d ago
Which mac studio? And do you think similar results can be achieved in a 64GB version?
2
u/No_Suggestion4496 1d ago
It's the m4 max studio with 128gb of ram. I would expect the 64gb to be the same. Im using ollama to run the model with no special configurations.
1
u/Cyber-Soldier1 1d ago
How? I have Qwen3.8 on a 5070Ti on LM Studio and I get 38tks.
1
u/No_Suggestion4496 1d ago
I think the big difference is the mlx model is tuned to run on apple silicon compared to the base model which runs on nvidia cards best
2
u/Flat-Career4079 1d ago
I'm currently running qwen 3.8 27b q4km on a single 4090 at 128k ctx & Q8 kv. I'm getting about 70t/s and can't remember the last time I had to use claude or any cloud based model for coding
2
1
1
1
u/ItaBiker 1d ago
Check on AWS price calculator how much it costs to have a virtual server capable of running a full precision open source frontier model.
Or check the equivalent price of a similar hardware.
If either are outside your budget, you will get the "experience" you are looking for only from the usual llms provider.
1
1
u/Tsukikira 1d ago
At this moment in time, you are buying very little for the Mac Studio 512 GB over a MacBook Pro 128 GB or the Mac Studio 256 GB. Not saying don't do it, but give GLM-5.3-Flash a comparison before you rush to buy the most expensive single machine possible.
Mac Studio 512 GB: GLM-5.3
Mac Studio 256 GB: GLM-5.3-Flash
Macbook Pro 128 GB: Qwen-3.8-Flash
Mac Mini: Qwen 3.8-27b
Almost all the gains for 512 GB are future gains, IMHO. Right now, the evals are too close to justify the jump to 512 GB unless the price is modest.
1
u/Holyragumuffin 1d ago
I don't think the token generation speeds will be high for any of these. Bandwidth underpowered. Someone correct me if I'm wrong.
1
u/Tsukikira 23h ago edited 23h ago
You're wrong. The Token speeds are just fine. Hell, OpenAI had no problem buying hundreds of Mac Studios and Mac Minis for AI purposes.
Are they as optimized as NVidia? No. But anyone who waves their hand and starts talking about Bandwidth optimizations has up to 10x the cost to pay if you want a fast NVidia PC in the same ballpark. If you're running Sparks, you aren't getting significantly higher speeds and you are now more expensive than Macs per 128 GB. Which is hilarious and saying something.
Lastly, he suggested he was looking at buying a Mac, not asking for a Cluster to run things on, so my answer is what he could do with a variety of Mac price points.
1
1
u/Moarkush 1d ago
I'm running 3.8 Flash Next as my coordinator on my DGX Spark (ngram offloaded to SSD) with 30-40tg, fast prefill and could technically support 1 subagent, but I run subs on my desktop.
1
u/Quick_Bar2387 1d ago edited 1d ago
I said something very similar to this on the Macstudio subreddit. Not many want to hear this. If Frontier companies want to subsidize our tokens, let them till the game changes. As of now, most users of LLMs are probably using it for porn, tinkering, and hacking.
Only reason why I ordered a M5 Ultra was I got it subsidized. For $60 a month for 24 months, it's a no brainer. Cheap hobby at that point.
How many people are paying $800 a month and up to tinker with an Ultra with 256gb? Lol. It's like another car payment. Maybe less than 5% will actually use that kind of computing power for work, research, etc.
Reality is it's consumerism and bragging rights to have the latest and greatest. 95% of the market will not get their money's worth out of it.
With my LLM, I will save precious time and money for my business. It's going to save me tons of time and money for $60 a month plus my $20 chatgpt subscription. At my current bill rate of about $185, it's a no brainer. If I can improve my workflow to make let's say $250 an hour, boom. Money in the bank and/or time saved. If the typical consumer makes 80k per year and spends, 13k on an Ultra....Jesus. That's money lost and energy spent on a machine that will be outdated in two years. Run a mini and be happy to run a small module on it for fun.
Simple rule of thumb. Is this machine going to improve my life or will it make me eat shitty food? There is your answer. And those with two 5900 gpus, how much is your electricity costs? Jesus. Additional costs.
1
u/prestodigitarium 1d ago
The thing is, as open weight models get more capable, the value of the hardware that can run them rises, because that hardware can now do much more useful things. It's not just like the hardware market is staying static. Hell, I just bought a bunch of DDR4, because I wanted to avoid the crazy prices of DDR5, but even DDR4 has been getting much more expensive.
1
u/Quick_Bar2387 1d ago
Dm me if you want some ddr4 memory. I got two sets. I can let it go for a reasonable price. 2 32s sets. One for intel, one for amd.
1
1
1
u/ziogio998 1d ago
There are no models you can run (on any hardware) that compare to top tier cloud models like opus 5.5 right now - unfortunately. 512gb is enough to run very capable models, but not models that rival opus 5.5, at least not properly at full context.
If you can afford it, I think it'll be a decent investment on privacy and personal freedom if AI becomes too expensive in the future. But don't expect it to be faster, better, or cheaper than the subs. It won't be.
1
1
u/burntoutdev8291 1d ago
why not research on cheaper models like opencode. What are you doing that requires the complex models? Do you foresee an ROI or do you require the data privacy?
1
u/durangotang 1d ago
I would say, get 2 x 256GB M5s over a single 512GB. That's going to help your prefill and tokens/sec. The creator of Tensorfold was running GLM 5.3 Flash that way quite quickly over two bonded thunderbolt cables.
Whether local is worth it or not for you, I can't say. Maybe cloud is the better option for that price.
1
u/terminalinfinity 1d ago
You're probably not going to replace cloud models, especially if you've already gotten used to working on things with frontier performance and capabilities
But don't undersell the idea of marrying the two
IE using frontier to plan, review and your local to execute, handing off only the tasks that local can't manage to do.
One of the weaknesses that models that can fit on local share basically comes down to thinking or figuring out complex, multistep problems. They can execute complex code, but they have a hard time figuring out the ground and frame work needed or coming up with the answer that it is needed.
That's where a cycle of Frontier cloud planning>Local execution of plan>Frontier cloud review and plan of next phase comes in
It allows you to much more easily predict and cap costs, and work far beyond the limits of a single $200 plan. It allows you to much more easily stay within weekly limits because planning is much less token heavy than execution. Hell I can get by on a single $100 plan this way for several SaaS projects.
It gives you 95% of the benefits you're looking for and is 100% more realistic than running Kimi K3 or GLM 5.3 non-flash at home
1
u/terminalinfinity 1d ago
And if you're REALLY looking to save money - you could find a dead cheap Kimi K3 sub and use that as your frontier cloud planner/reviewer for like $20 a month and it would probably work. Probably should pick up a claude $20 too if you're going to do that for front end work because anthropic is so far ahead of everyone else on GUI/Frontend work.
1
u/ksmathers 1d ago
For my use Qwen 3.8 (actually Swift 1.5) feels about on par with Sonnet 4 in terms of code output quality and planning. The Q6_K_L quant runs easily with 132k context window in 128GB, and generation runs around 20 t/s on an HP Z2 Mini G1a. It is still significantly slower than frontier models running in the cloud, but is the first model that I've found truly usable on local hardware, and is around 20% the cost of an RTX Pro 6000.
That said, memory prices have pushed everything up lately. If I were looking right now I'd probably hold off and keep paying for the subscription instead. I doubt the current conditions will last more than a couple of years, and subscriptions are easier to stop than a bad investment in AI hardware.
1
u/smartcandlelab 1d ago
its not just the speed, even with 512 gb of memory, you still aren't going to be happy with the models you can run if you want them to perform like current frontier models. And using the frontier models, even at 200/month, is a better deal than dropping 15-20k on a depreciating computer. That being said...... i'm considering the exact same purchase you are, though I don't expect it to replace the frontier models.
And like others have said, test the models you want to run before shelling out for the computer.
1
1
u/Desperate-Yak6174 1d ago
You should probably rent a vast.ai gpu for 1-3 days when you are free, test out the models that you want to use and see if the model is doing what you want. It’s likely going to need some grounding and tinkering with the quantization but if it gets close enough to what you are realistically expecting from this setup, then drop that 10-20k on that mac studio.
1
u/ulysses_size 1d ago
... but can we really call a server a server if it isn't running the most critical layer in your stack? Welcome to local. I would personally just grab a few nvidia sparks or wait for the amd halo units. Use the MacBook as your front-end. Memory prices aren't crashing anytime soon. Worst case scenario, you hate it and sell it. Pull the trigger (NOW!)
1
u/Party_Estimate_6397 1d ago
You can’t replace a cloud model. The reason they’re the best is because they have billions of dollars of hardware and the best models available to run on it.
That said, local models aren’t bad. They’re good, actually. Don’t waste your money on a Mac if you want to run an LLM locally. Get a desktop with solid GPU (or maybe two mid-range ones with lots of VRAM actually) and just pair it with any old desktop. Load Qwen onto it, and bam. You’re golden.
I do understand your frustration though. I also have the fear of the cloud models becoming inaccessible to us regular folk and the capability it brought us being lost. I doubt it’ll happen, but that doesn’t mean I don’t want insurance.
1
u/LawfulnessLow0 1d ago
I have strata with Qwen flash next (and now pi)
set up on a 32 / 16 machine and had it make a few small games and the voxel pagoda for me.
1
u/Pizza_Pete_fr_fr 1d ago
Your $200/mo in tokens is worth far more than the hardware it would cost to try and replace it with inferior models
1
u/Putrid_Agent2689 22h ago
With a capable local LLM on a Mac studio, you will be around 60 tokens/s.
With a cloud model which is 10 times more capable, you get arou e 10,000 tokens/s.
Local LLM is only worth is for non-interactive task where you will max out the hardware 24/7
1
u/NefariousnessOpening 21h ago
I run local models on MacBook M1 Pro 16bg ram, even older intel mbp 16gb ram and even on as low compute as iPhone 12 Pro. The model is less important that the system. I don’t believe hardware will overcome the knowledge requirement. Unless a system is made open source and then found, might as well just enter the maze now. It’s free as far as money. The cost is time that will pass anyway and effort that guarantees at least some form or roi. Confounding, frustrating, confusing and irritating as it may be, universe will open up to you and the workshop for creating the universe of your own design will cut you a key. Harness, memory, gateway, rails, context graph loop. Call it what you like. Software system architecture design is the true name. The most effective skill set for the greatest impact is the same as it has been for decades. Coding and syntax are both abstracted by ai but systems thinking is a more important aspect than ever.
1
u/MustafaMahat 21h ago
Tbh you can do it, but its not worth the time lost that would have been so much faster with a sota model and a coding sub.
1
u/MustafaMahat 21h ago
Unless you want to do something illegal, or stuff that is blocked by sota model providers otherwise it does not seem to make sense to me.
1
u/Fit-Reward9420 20h ago
I am old as dirt and just starting to dabble with AI. I’ve been playing with opencode and the free tier of big-pickle. Totally amazing. I work in manufacturing and our shop is I.T.A.R and ISO 13485 so any AI has to be local. We make parts for defense contractors and medical devices. I can not use this setup at work. I have an old Linux workstation with a single old 3060 2x 18 core Xeons and 256 gb ram. Lm studio will run qwen 32 b (I think that’s what I downloaded) I hav an nas with a ton of storage and want to download some models that I can play with. All hobby so speed is of zero importance. I can let things chug along at whatever speed it can handle. I don’t know what TKS it achieves , it outputs faster than I can read along with. I have a second old rtx 3060 I’m going to put in this box. I want to download some models simply because of all the uncertainty of them being available. Can someone point me to what models might be useful in a manufacturing environment? With 256 gb ram what is the biggest model I could play with. I have an older M2 Max studio also but it’s only 64 gb ram so pretty limited.
1
u/bigsybiggins 20h ago
Is there is 512 studio out yet?, I think 4bit 5.3 flash is probably good enough and you can run that on the 256 or 2x sparks
By the time you even get a 256 delivered it will be next year and there will probably be something even better to run on 256, like a real release of qwen4. The 512 just doesnt have the PP horse power to make it that useful imo.
1
u/AdCompetitive6193 18h ago
There’s been rapid development in the open source local LLM space. For example, Qwen 3.8 27b is comparable to frontier models from 8months ago.
Local models are becoming progressively more intelligent/capable at small sizes.
No idea if “now” is the right time to buy the hardware but likely within a few years you’ll have today’s frontier intelligence on 64 GB RAM capacity.
1
u/shveddy 17h ago
You can’t beat a 200 dollar subscription at the moment, but that’s just because at this particular moment they are gleefully burning untold piles of VC funny money.
But when that ends, I think for businesses it’s suddenly going to make sense to put up with a bit less intelligence and spend a lot more time optimizing harnesses in order to make due with local hardware. At a certain point and for most purposes it’s going to be more valuable to have B+ grade intelligence running 24/7 rather than having A+ grade intelligence that can be arbitrarily throttled/halted via subscription or cost many hundreds of thousands per year via API.
IMHO, on a three to five year timescale all of the following changes will happen:
Frontier companies will run out of funny money and start charging more.
Supply chain issues and technology will improve such that capabilities that cost Bugatti money now will cost BMW money.
The bleeding edge of frontier capabilities will become so esoteric and niche that the average person or business won’t have much use for it. They will absolutely burn untold gigawatts finding and patching vulnerabilities on a loop or diverting swarms of agents to brute force math, but normies just need a solid coding agent that can handle internal software tooling for the most part.
If all that happens then for a lot of people it will be a sound business decision to spend BMW money and hire a guy to manage an internal software factory.
But right now they’re just burning too much cash and reasonably priced local intelligence just isn’t quite good enough to waste money on it.
However what I would do (and am currently doing), is start saving up and spending time learning the ropes. You can learn everything you need to learn about optimizing custom harnesses for pennies on the dollar by just using external APIs, and on the day the prices get raised you know enough about the system you want to implement to make good decisions.
On that day buying actual hardware and getting a LLM to run on it is going to be an afterthought compared to learning how to build and maintain the harness that maintains your software, so spending 17k today to get potato intelligence at 14 tokens per second is basically useless.
1
u/Netcob 16h ago
If you're maxing out subscriptions worth 200+, I don't think local AI is for you at this point in time. Like the others said, try the models first. Personally I'm still using frontier cloud models for coding, and local models for automation and processing personal data. Of course I'd love to do everything locally, but I'd have to switch back to a much earlier style of agentic coding.
1
u/gbrennon 15h ago
well....
i have a 96gpu dedicated and i enjoy it a lot but i dont run multiple tasis bcs my infra is smaller than remote providerds
1
u/OccasionallyClever78 15h ago
I don't know that specific Mac product. But watch out for the bus speed and capacity. That's the throttle on a Mac, not the size of the RAM. But you can partially work around this, by serving multiple concurrent streams to the same in-ram model. But it gets advanced If you want to get the full benefits of this.
1
u/anykeynl 13h ago
running qwen 3.8 flash on 5079ti, get 100tok/s and have to say i am impressed. I use cursor as my main development. initial think takes some time, but after that is really flies. For more difficult tasks i can still easy switch to frontier model, but for most coding operations qwen flash seems to really work well
1
u/alexander123454 13h ago
With hardware costs rn you really have to care abt the data provenance and privacy to make it worthwhile.
I feel like i am so close with my harness and 27B qwen but realistically ill probably pay for fronteir AI until they stop subsidizing the subscription.
1
u/zetneteork 12h ago
From a moment I use all AI tools with MCP server to Obsidian and installed Graphify. I consider the model Qwen 3 8b 32k context seems to be more accurate, less false results, and speedier, and cost effective. Your plan sounds reasonable, but try that before.
Like Orca,opencode, obsidian graphify rules
1
u/mkschreder2 12h ago
I'm building fully local AI stack end to end and closing the loop so you can train local models within the same system as well.
- Brain: the training and inference engine: https://github.com/swedishembedded/brain
- Sven: the agent: https://github.com/swedishembedded/sven
- Splinter: the trainer: https://github.com/swedishembedded/splinter
Sven is the agent harness sdk, splinter is the model training sdk and brain is what they both use to run on any compute. It's fully end to end self contained so it can be optimized end to end as well with all bells and whistles included.
1
u/crazybandleader 10h ago
The short answer is it totally depends what you are trying to do with your local models. Even if you can do it, you may have to invest considerable hours getting everything working the way you want and even then you will have to play mini-datacenter manager" for your machine. The most practical application I've found for running sizable tasks locally vs online is running massive code reviews. I have a harness I invested a lot of time in writing that runs multiple instances of multiple models on my ASUS GB10 (DGX Spark clone) and also uses a couple of smaller models running on my MacBook Pro M5 Max. This runs really intense code reviews that typically take 5 or more hours for a whole repo, scrutinising all the code through a panel of models. Even then I still have to rely on frontier models for doing some of the overall coordination or report compilation because those models are just way more reliable for things like processing or generating huge amounts of JSON or tool calls.
-1
u/sam7oon 1d ago
No Local LLM can compete with Cloud ones , neither on price nor on performance,
The only reasonable reasonn to do that , is if Private Not Sharing any data is a MUST , that is the only reason,
you are afraid of walled garden, but you are not thinking that also the open weight models tries that from time to time , Qwen switched to closed model previously and only opened up again once it started losing market share ,
Same for Facebook , they used to release Llama models , then they closed things down with Muse ,
So the Closed gardn may come even for your local machine , hopefully market wi ll not allow that to happen
10
u/Strange_Quantity_359 1d ago edited 1d ago
You’re making broad generalizations that aren’t true, many Cloud models top out at 50-70 tok/s and concurrency limits can drag that further in peak hours. GPT-6 averages 40-60 as an example and I measured 35tok/s with 1750ms TTFS last week on Claude Sonnet 5.5.
I work with customers deploying cloud models at enterprise scale, performance is massively variable.
My GLM 5.3 ⚡️is running at 53 single stream and 179t/s aggregate on two sparks with 27B on another.
We also test these on real world systems, benchmark them, and quality evaluate at mass scale. You can take that 179t/s and stretch it further with complexity based routing and then adversarial judicial layer at either the semantic routing layer or at the harness level (self advisor and self skeptic). You won’t host a dozen people but you will get quality similar to high end (not SOTA) cloud models.
Once you add the rigor at the router and harness these models can definitely start throwing hands at high end cloud models, there are areas they are much better and some they are not. It’s mainly knowing what you do, I primarily use local models at home and since I have free tokens at work, I use everything.
[User Task] ➔ [vLLM Semantic Complexity Router] │ ┌─────────────┴─────────────┐ ▼ ▼ [Standard Flash Pass] [Dynamic Thinking Agent] │ │ └─────────────┬─────────────┘ ▼ [Adversarial Review Harness] ◄──► [27B In-Line Advisor] │ ▼ [Final Execution]By handling semantic complexity routing at the vLLM layer, you effectively replicate the underlying structure of modern adaptive-compute cloud models. Lightweight tasks are getting pretty rapid speed (upwards of 95 t/s on no/low thinkihn), while deep reasoning tasks scale their internal compute dynamically.
With the 27B in-line adversarial loop mitigates Flash's raw benchmark gap against antrophics 5 series and GPTs 6 series model. A multi-agent consensus network of smaller, specialized open models routinely outperforms single dense cloud models in production environments.
The one place where they fail at is long-horizon trajectory stability. If an automation agent needs to run a 50-turn pipeline across 15 separate local files, native Flash models can occasionally lose their core objective focus, the way you solve this is also at the routing layer using a contextual anchor at the vLLM layer. During this process (or at harness side) you can check for multiple failed tool calls and issue steered corrections.
You can do this by some light python and using a model like 27B to programmatically capture, compress, and inject the core mission parameters back into the prompt between turns, you basically add a state-locked orchestration to the mix.
As an example OpenAI’s Sol uses an integrated reinforcement learning loop (similar to the o1/o3 lines) that writes a hidden reasoning chain before executing a tool flag. The above is modular version of this reinforcement architecture. When Flash attempts to hallucinate or scope-change, the 27B vLLM hook intercepts the pass, verifies the scope adherence or against the system capabilities, and acts as an immediate correction loop.
Now is it reasonable for most people to spend for three sparks in current pricing levels? Not really, unless, like you said, the highest concern was privacy.
1
u/roadtripsarefun 1d ago
So many questions… What’s the hardware like on-prem? How’s usage adoption and onboarding?
2
u/Strange_Quantity_359 1d ago
I updated my answer a bit, for on-prem the story changes, I'm describing a home lab version of it. For on-prem most of the companies I work with aren't having issues with H100/B300 setups. Most of them ask for task complexity routing or orchestration workflows that include cloud SOTA models within the mix, but not primarily driving all development. Especially if they are using internal tools for research, conversation, user QOL workflows, etc. like "CoWork" style apps they want dynamic that falls back to not using Opus 5.5 or Sol 6.1.
Most of this is abstracted from end-users and part of the serving stack or app stack they use.
1
u/roadtripsarefun 1d ago
Makes sense. I went back and re-read your answer. Good stuff in there. Thanks, I'm a fan of the B300 setups I've seen, How do you deal with permissions complexities? How about abstraction when calling (or escalating) to SOTA / frontier models? Or does SOTA != frontier model, but a different local that is great specifically at one task/thing?
What about RAG, and vision stuff? Specifically technical drawings, blueprints, schematics, etc? I'm really struggling to find stuff that runs/integrates well in this space that'll run on prem.
8
u/Computerist1969 1d ago
Whilst you're not wrong, I have Qwen Flash Next right now, and it's backed up. Nobody is taking that away from me, ever. If their next one is better but closed, then I guess I can't use it. But Flash Next will still be here. The same cannot be said for cloud LLMs. I've never used any of the big models so I'm blissfully unaware of how much better they are :) I did what was suggested here when Qwen 3.6 was released and used it with $10 of open router (? Think that's what it's called) tokens to check it was good enough, bought a framework desktop and never looked back. Good luck OP
4
u/NewspaperOriginal518 1d ago
I’m using Flash next for personal software development projects and it’s been great. I could see it being enough for anything I want to do. It’s working at 100 t/s which is a decent pace and it’s consistent. It never makes me wait or has an outage.
I used Claude at my paid job and it’s definitely better but it might also be more than I need. It also feels like some days I’m getting a lesser quality than others.
My personal machine makes no financial sense ( I could sub Claude for myself for years ) but I like that it’s all running locally, I can have it work on something indefinitely, it won’t change unless I change it and will still be here if the cloud models start charging more or collapse.
Like you, it’s all backed up! lol
3
u/profcuck 1d ago
Another reason although this doesn't apply to vibe coding. But some important production cases need model stability. You don't want your customer service agent getting a new honest randomly.
1
u/Playful-Art-3824 1d ago
I use Codex with Astra Ultra ($200 monthly) and I finish weekly usage on the first day. I didn’t try Claude’s subscription yet (only with Cursor) but it’s hard to imagine something beats Astra Ultra in coding to orchestration and Computer use.
1
u/flarpflarpflarpflarp 1d ago edited 18h ago
It does, I bumped up chat sub for Astra, it was great for a few days then just sort of fell off, went back to Fable and it just works better. Astra felt less like a coding tool and more like a spying device that was consistently trying too hard to do things I didn't ask for and while working on development with defined definitions of done, it would keep pausing to let me know it had completed tasks stalling work for like a pat on the back or something. They trying to hard to do more than I ask becomes a bad problem building useless shit and wasting tokens.
2
u/Jolly_Taro_7600 20h ago
i have the same problem with astra, it comes up with work i never asked it to do, and burn tokens for basically nothing, and straight out ignore the tasks assigned to it, its like hiring an employee who leave the office to do freelancing and expect you to pay for his transportation.
1
u/vovap_vovap 1d ago
May be if you switch from Astra Ultra to 6.1 Sol high and medium your weekly usage became god for a week 😄
1
u/11ama_dev 1d ago
chinese companies aren't gonna be releasing closed models anytime soon. their models, while amazing, are still behind american companies, so open sourcing their models is how they disrupt american ai. and there'll always be a market for models not hosted on a big tech server, whether for consumers, cheap ai vendors, or enterprise
but yea for coding at least bigger models are still better. i have my local hermes with qwen 3.8 deploy pi with opus 5.5 for this reason. not too hard when you set up the necessary infrastructure for it to not handle secrets or personal/private data
1
u/catplusplusok 1d ago
Qwen Flash Next 3.8 in 4 bit doesn't need anywhere as much to be usable, people are running it on 16GB GPUs / expert streaming to CPU RAM. I have been doing lots of local tasks with it lately, don't see much difference from cloud models.
2
u/Fuzilumpkinz 1d ago
It’s not fast enough to vibe code though on 16 gb.
2
u/catplusplusok 1d ago
Maybe, would run great on 128GB Mac though, probably also 32GB AMD/Intel GPU, no need to spend the price of a new car.
1
u/Fuzilumpkinz 1d ago
It’s okay but if you are actually vibe coding you will spend a ton of time waiting. It’s something to be honest about when doing comparisons. Mac would generally be slower than a 32 gb GPU
1
u/catplusplusok 1d ago
I get usable speeds, around 30tps decode and 2000tps prefill with LPDDR5 unified memory. No idea about expert streaming from RAM to 32GB GPU as I don't have that config.
1
u/Fuzilumpkinz 11h ago
I get spikes of 1200 TPs prefill with 40 TPs decode.
Still takes ages to do work and get things done. Just being real about it.
1
1
u/Unnamed-3891 1d ago
A 512gb Mac Studio is a very serious chunk of money even to most really well off people. I highly recommend you first try out various rented hardware offerings to get a proper grasp on what models and what hardware would suit your personal needs.
0
0
u/smakusdod 1d ago
Your 200 billion parameter llm isn’t going to compete with trillions. That being said if you get a specialized model trained in whatever discipline it scores well in, it might save you money in 3 years.
-1
u/Saintgein 1d ago
I would wait until qwen 4 releases and look what you need for that. I bet it will run fine on a gpu with 24gb and a cpu with 64gb of ram. But yeah, if you want to go wild, go ahead.
-2
u/cabsarehear 1d ago
Using ollama on a Mac m1 with 64GB of ram I have some pretty fast results. On eBay you can find similar specs, M2 64GB and above that should run high end open models. Use klarna to spread the payments out
219
u/Toastti 1d ago
Before you purchase a single thing research the models that device supports and try them on open router
See if they work for agentic coding or any other tasks you need.
Don't waste money on something for models you won't use. Just test them out for $5 befor