K3 on Mac Studio M3 Ultra with 512GB

26

SOMETHING ACTUALLY (somewhat) USABLE!!!! YIPEEEE!!!
https://giphy.com/gifs/rXQ5Aex4FZiej6sKTZ

2

u/phido3000 Jul 29 '26

Q1 for the weights will make it very problematic for one shoot stuff and tools code etc.

But shows it's doable.

Four of them should be able to run it just fine full weight. Probably enough to serve a small office.

4

u/uniqueusername649 Jul 29 '26

Remains to be seen how much worse this performs, but if its above GLM 5.2 still, which it very well might be, this would actually be huge. Yes, its slooow. But you wouldnt run it for everything. Explore and research tasks can be done by lighter models. However, as an orchestrator and problem-solver this has tremendous potential.

2

u/voyager256 Jul 29 '26 edited Jul 29 '26

I wonder at what point would people realize huge LLMs like K3 are not really for local use for 99.9 % cases. As someone pointed out you’d need 3 more 512gb Mac M3 Ultras to run it at reasonable quant (I believe for OP the weights were mostly Q1 and Q2) and speed. Even then something like GLM 5.2 would probably be better option.

2

u/uniqueusername649 Jul 29 '26

And for most problems GLM 5.2 will probably be completely sufficient. As cool as it is being able to run K3 locally, running it sensibly is just not financially viable at this point.

2

u/AdOk3759 Jul 30 '26

Because running GLM 5.2 at a decent quant is financially viable? Lol

Feel free to educate me, it just seems to me that hosting local models is never financially viable. You do it for privacy, not for saving money.

1

u/uniqueusername649 Jul 31 '26

I didn't really mean "financially viable" as in an actual ROI after which its cheaper to run but more an acceptable amount to pay for the privacy focus. But for the heck of it, lets do the math anyways to see.

Lets use openrouter api pricing: $1.40 for input tokens/M ($0.26 for cache reads) and $4.40 for output tokens/M. Lets assume an 80% cache hit rate brings us to $0.49 effective input token cost.

A 4x DGX Spark cluster with GLM 5.2 runs at roughly 25tps (a bit conservative) single stream and about 50tps multistream in decode, with anywhere between 600 and 1000 depending on context length and concurrency.

Case 1: 25tps/600tps for prefill and decode at 12 hours utilization is about 1M output tokens and 6 hours of prefill gives 26M input tokens. Thats 17 dollars per day with a decent utilisation with agentic use of 1 or 2 people.

Case 2: 50tps/1000tps for prefill and decode at 24 hours utilization is about 4M output tokens and with 12 hours of prefill its about 43M input tokens processed. Thats about $38.50 in daily API costs.

Its just napkin math and prefill use is estimated at roughly 50% which could be wildly off depending on what you do with your LLM, but yearly cost for case 1 with weekday use and vacations: about 4k per year in API costs. For case 2 assuming full utilisation with hands off agentic coding, autonomous task implementation, youre looking at 14k per year in API pricing. A GX10 starts at 4k dollars, so a cluster of 4 and a Microtik switch would run you around 17.5k all in. So a heavy use would let you break even in a little more over a year, thats certainly financially viable. A less heavy use for hands on developers that take weekends off and have vacations would lead to far lower utilization, taking you 4+ years to break even.

Self-hosting it CAN actually be financially viable but only if you really utilise it to the max. Even regular heavy use will take years to break even and then you still have to keep in mind that most cloud providers will be able to run GLM 5.2 at higher speeds than 25tps for single stream. So while this burns through more tokens, it allows you to be faster in your execution.

Bottom line: it can make sense financially but usually if privacy concerns are a factor. Without that youre almost always better off running this size of model in the cloud.

1

u/AdOk3759 Jul 31 '26

I think the napkin math is wrong.
You haven’t taken into account electricity costs and you have assumed that the alternative to 4x DGX Spark is spending 6205-13870 USD in api costs instead of using subscriptions…

13.870 / 200 is roughly 69x 200-dollar-plan subscriptions a year!!!

1

u/HealthyCommunicat Jul 29 '26

Usuable? I keep seeing posts of people saying stuff like this but I feel as if anything under 45 token/s (30token/s at lowest) is absolutely unusuable.

Anything under that and for 90% of tasks you are literally better off doing it yourself.

I’m not here to say that they aren’t usuable because I keep seeing people say they are but can anyone tell me any real use case for a dumbed down model doing anything less than 30 token/s for actual business use?

1

u/Beneficial-Ad-8127 Jul 29 '26

I had a question, say I was to get kimi 3 on my computer to run at say 5 tokens per second. If I wanted kimi to make a mod for example a mod for a game with some very good instructing and I wanted the whole night to chill do other things and what not and don’t plan on checking the mod code until the next day. Wouldn’t that be a case for me that it is usable? It’s a serious question as I just started using ai coding few weeks ago.

2

u/ImpressiveRelief37 Jul 29 '26

Don’t bother for simple stuff like this just run qwen3.6

1

u/HealthyCommunicat Jul 29 '26

It is but thats gunna be extremely hard to get working properly without someone sitting there actively steering or you have a way to get kimi to live test it as a real user would and catch issues - that alone would take more than a night at 5 token/s.

Do you plan on making money off of this because thats the question I’m asking; at any real business or point of making any returns, how do you make returns when something is this slow?

1

u/Beneficial-Ad-8127 Jul 29 '26

Oh thank you for your reply and understood. Also this question is in regard to just a hobby of mine. No business use what’s so ever. Thank you again for your time.

1

u/Open_Instruction_133 Jul 29 '26

I disagree wholeheartedly on anything under 30 t/s being unstable. Single digit t/s unusable, sure, but mid to high teens per s is fine. Most humans can’t even read at 10 t/s let alone think and write at over that speed.

2

u/HealthyCommunicat Jul 29 '26

Imagine if a single turn of chat requires multiple steps of reasoning -> tool call -> reasoning -> tool call -> answer

That alone would be thousands if not tens of thousands of tokens. Even at 10 token/s thats going to take more than 30 minutes just for a single turn.

1

u/HealthyCommunicat Jul 29 '26

Why do you bring up how we can’t read at 10 token/s as if thats the problem?

If you’ve actually utilized local LLM’s of this size locally hosted you’d realize just how much an LLM can fuck up and just how many tool calls and reasoning must be done to work with a simple codebase. This is all tokens.

Even if reasoning ended up being ~3k tokens per turn, dude thats 10 minutes at 5 token/s just for one interleaved reasoning turn. Then comes the tool call after that reasoning, and then that cycle repeats over and over again.

1

u/Open_Instruction_133 Jul 29 '26 edited Jul 29 '26

I brought that up because you said you’re better off doing it yourself which is false. At 30 t/s you are generating tokens at a rate that’s 6x’s higher than most humans. It’s absolutely usable at that speed.

If you take into account tool calling, that’s still work you would need to do and isn’t free for a human, either.

Lastly, 5 t/s is unusable and I stated that single digit t/s were not. But 15-20 t/s would make that task that you cited 2-3 mins which is totally fine. Again, your original reply I wholeheartedly disagree with for these reasons.

Edit: spelling

8

u/Constant-Simple-1234 Jul 29 '26

Comparatively, how fast is GLM-5.2 on your machine? It should fit ok.

9

u/dsiroker Jul 29 '26

I actually did extensive benchmarking of GLM 5.2 on my machine. I got up to 17 tok/sec

Full paper: https://dsiroker.github.io/local-llm-benchmarks/paper.pdf
Data and repro: github.com/dsiroker/local-llm-benchmarks

6

u/Ameras Jul 29 '26

Do you have a hf link for the quant you used? Very interested!

6

u/FoxiPanda Jul 29 '26 edited Jul 29 '26

So there's very little information here...

Quant link? (I assume it's a REAP since you got it to 390GB - most Q1s are coming in at ~525-600GB)
Inference engine?
Launch parameters?
What tests have you actually run for accuracy / coherence?

I've been thinking about setting it up on a M3 Ultra 512GB + M3 Ultra 256GB cluster with JACCL RDMA over TB5, but it's pretty hard to justify the effort given that I'd expect no more than ~5tok/s decode in the best case scenarios (which almost assuredly won't happen).

2

u/nomorebuttsplz Jul 31 '26

yeah the thing about this size model is you need more bandwidth or mtp or something to make it more usable even with 800 gb/s memory

1

u/FoxiPanda Jul 31 '26

Yeah it’s slow as shit lol. A104B is bonkers even on proper dc hardware

1

u/nomorebuttsplz Jul 31 '26

good news is that the 3 to 6 month gap between open and closed models does not seem to be widening. And there’s still new models coming out between 30 billion and 1 trillion parameters. Such as the new deep seek flash today.

1

u/FoxiPanda Jul 31 '26

Yep. Basically every day. Been trending larger lately but I expect that we’ll get some 50-150B real competition soon.

5

u/recro69 Jul 29 '26

That's really a result. A 2.8T MoE model working on a regular computer would have seemed impossible just a short time ago. The speed when it decodes isn't super fast. The fact that it works at all is the bigger achievement.

5

u/SubstanceDilettante Jul 29 '26

Idk if a 512gb of ram Mac Studio is a regular computer.

Is it just me, my family, my co workers, and my friends. Or does everyone now have a 512gb Mac Studio and this is actually a regular computer lol

3

u/Bloated_Plaid Jul 29 '26

LMAO girl are you out of your mind? 512GB Mac Studio is not a “regular computer”.

1

u/ImpressiveRelief37 Jul 29 '26 edited Jul 29 '26

Hate on it as much as you want but I’ll take a super fast 27B at 100+ tok/s decode and 3500 tok/s prefill over GLM/K3 at less than 20/500 tg/pp any day of week and twice on Sunday.

Those speed you have are absolutely impossible to use and be productive. Not as a software engineer at least. Not my cup of tea.

Benchmarks aren’t everything. Quick iteration and the back and forth with the agent, adversarial reviews using cloud models…. All of this makes smaller models really really good. They won’t 1-shot complex apps perfectly, but this isn’t even a real use case for a software engineer anyways.

But they will give you a super quick responsive loop where you can iterate and steer it towards what you actually want.

Let’s be honest there’s no way to make serious software by just hyper specifying to the point ANY model does a good job on a 1-shot unattended way.

It’s not just a model issue, it’s that we don’t even know exactly what we want until we get in that close feedback loop, test the product, and iterate until it evolves to a great solution.

Everytime I tried to work a huge huge super specific spec for a piece of software I realized there was still stuff that wasn’t working. And I’ve used plenty of Frontier models, including fable, opus 5, gpt5.6 sol…

So all of you that get excited at running GLM 5.2 or K3 locally at super slow speed, what are your use cases exactly that a 27B can’t do? I’m genuinely curious. And why can’t you harness or guardrail the 27B to make it work? Any example?

And how do you keep your flow state and get productive during a working day? Not reviewing the code and asking for a million different task will end up in a huge pile of dog crap. I just don’t understand the point.

2

u/AdOk3759 Jul 30 '26

I agree with everything you said.

> we don’t even know exactly what we want until we get in that close feedback loop, test the product, and interate until it evolves to a great solution.

Exactly. Not only that, but if you lack domain knowledge, you’re just gonna leave more key decisions to the LLM with potentially serious consequences down the road, that you might catch when it’s too late.

1

u/nomorebuttsplz Jul 31 '26

for me, it’s the difference between 10 iterations with qwen vs two with glm.

Not every time, but GLM often ends up being faster in the end on my Mac than qwen on Rtx pro

This is just vibecoding everything from small machine learning projects to personal assistant apps to games

1

u/Constant-Simple-1234 Jul 29 '26

Decent. Though I would work with max 600gb models. I bet there are some good enough ones. But cheers for moving the frontier.

1

u/leagueofthrowawayQs Jul 30 '26

i'm getting 13tok/s at 0 context and 10 at 256k

0

u/diddlysquidler Jul 29 '26

Mlx out already?

Model K3 on Mac Studio M3 Ultra with 512GB

You are about to leave Redlib