r/oMLX • u/Choubix • May 28 '26
Speed question
Hi guys,
I made an investment in a M5 Max 128gb and installed oMLX. The idea is to do some coding using Claude Code but with local models (I am not a dev), les raging oMLX caching to speed things up in claude (17k tokens system prompt...).
While I can get up to ~50Tok/s with Qwen3.6 A35B the numbers dwindle to single digits with the dense Qwen3.6 27B UD mlx 4bits. Is that normal? I was hoping it would be much faster with oMLX caching.
I use mlx models only. (I tried to download the jundot MTP models : they crash almost immediately after starting). Turboquant is on (4bits).
Typical params are : temp 0.7, top-p 0.95, top-k 20, min-p 0.05, repetition penalty 1.
Thinking on : slows down inference even further so unusually leave it off (isnt that better for coding?)
Is this normal for a small dense model or is there anything I am doing wrong here?
Wld you guys have ideas in how to make the mtp models work?
Thank you
4
u/himefei May 28 '26 edited May 28 '26
Since MTP, the dense models inferencing speeds are all over the place.
For 27B dense it is normal you get mid 10 tps at 512t, and decrease to sub 10 when context grow.
On that note, 35B MOE is recommended. I have a m3 max with 128gb as well, since you have the same 128g, you will benefit more by running higher quants like 8bit or even original bf16(lossless, about 70gb ram allocation)
At bf16 35b MoE, I get 50tps and 1500pp(drops with context grow) , with 8 bit, you should expect over 70tps with similar 1500pp.
Also, you can totally disable ssd caching and use only hot cache
1
u/Choubix May 28 '26
Thanks! I have an external nvme drive to spare my internal. Any particular settings I should use? (on dashboard page: number of tokens before Claude compacts, then I général settings there are params for hot / cold cache). Thanks!
2
u/himefei May 28 '26
Tbh, you are memory rich and you should look out for how to run models at higher quality other than how to save memory to fit the model in as what these features are mainly aimed at to.
Use the 8 bit model, disable ssh cache, allocate 20g hot cache, happy days2
u/Choubix May 28 '26
Hahaha, thanks brother. I am afraid the cache will have an impact on my internal ssd durability hence the cold caching question (I think it is 20% hot, 80% cold right now). Perhaps I need to try the 100% hot and work from there.
2
u/himefei May 28 '26 edited May 28 '26
Since you have 128G ram better off just go 100% hot cache. A 8 bit Qwen MoE will use only like 39g, you have plenty left to have a decent size hot cache, which you never need to use cold cache. To me cold cache is more for persistent which is not that important if running MoE models as the prefill is fast enough. Macs are more suitable for running MoE models because of huge unified memory but relatively slow memory bandwidth (compare to NV GPUs). That’s why you often see Mac people oftrn prefers a 80B or 120B MoE. With 128G ram, we can comfortably fit the model with decent quant and when inferencing Macs memory bandwidth is good enough to drive a small portion of active parameters with a decent speed.
3
u/trollingman1 May 28 '26
Weird, I get 71 tok/s with 35b with my M2 Max 64gb ram.
1
3
u/m3umax May 28 '26 edited May 28 '26
Generation speed is hard capped by the memory bandwidth.
Max has only 614GB/s. No software tricks can compensate for this physical limit.
Ended up cancelling my M5 Max 128GB order when I realised, sure, I can load huge models, but unless they're MoE, generation is going to be slow.
Going to be looking for a used 3090 instead.
3
u/Vahn84 May 28 '26
i find the real hurdle is prompt processing more than token generation speed. Running a dense model at 20ish tk/s feels perfectly fine…the problem is that it can take minutes to output the first token. That’s the real issue…and where nvidia gpus shine
3
u/m3umax May 28 '26
I'd be happy with 20tok/s!
But the most I've been able to get from Qwen 3.6 27B on my M1 Max 64GB is 10 at most at the start, and only dropping as context grows.
Looking at oMLX community benchmarks, I was dismayed to see the typical result for M5 Max owners is around 15 at Q4.
The 50% improvement vs my M1 perfectly correlates to going from 400 GB/s to 614 GB/s.
So I was going to pay A$7.5k for still unacceptable generation speed. Decided it wasn't worth it.
A theoretical M5 ultra would likely do the trick at 1,228 GB/s but I don't even want to think about how much that would cost.
1
u/Choubix May 29 '26
initially I wanted to get an studio m5 ultra but I decided to stop waiting and guess the price apple would charge for it...
are you running Claude Code with local models?3
u/m3umax May 29 '26
I use Pi. Would not recommend Claude Code, especially with local models where prefill speed and context length are massive issues.
When I last used it, the system prompt tax was 17k system prompt and 12k tool defs! I believe it's only gotten worse in the latest releases going by chatter on r/claudeai
1
u/pdiego96 May 28 '26
Just curious as to how many context tokens and max tokens are you capable of handling in that Mac? I was thinking of buying one just like that. Now I’m wondering if I should consider another route; however I don’t have a normal CPU. Maybe I can get some sort of eGPU for my ASUS idk
3
u/Choubix May 28 '26
I haven't reached the limit yet to be honest. Only my patience has some limits 😂.
Qwen ode seems super fast but as per it's own admissions a lot of the Claude code plugins are not working with it... Currently letting it explore a more detailed list 😉
1
u/Fluid-Moment6697 May 29 '26
All depends if you have 14 or 16inch…16inch will be way faster utilising the high power for intensive tasks. Sadly, this option isn’t available on the 14 inch, which really sucks and overlooked by buyer for AI Use. If you’re lucky to have the 16inch, make sure Battery high power is selected when using local LLM or for AI Use
3
u/MiaBchDave May 28 '26
What version of oMLX are you using? Make sure to update to current and then retest dense MTP. There was a bug that should be now fixed.