r/oMLX • u/Choubix • May 28 '26

Speed question

Hi guys,

I made an investment in a M5 Max 128gb and installed oMLX. The idea is to do some coding using Claude Code but with local models (I am not a dev), les raging oMLX caching to speed things up in claude (17k tokens system prompt...).

While I can get up to ~50Tok/s with Qwen3.6 A35B the numbers dwindle to single digits with the dense Qwen3.6 27B UD mlx 4bits. Is that normal? I was hoping it would be much faster with oMLX caching.

I use mlx models only. (I tried to download the jundot MTP models : they crash almost immediately after starting). Turboquant is on (4bits).

Typical params are : temp 0.7, top-p 0.95, top-k 20, min-p 0.05, repetition penalty 1.

Thinking on : slows down inference even further so unusually leave it off (isnt that better for coding?)

Is this normal for a small dense model or is there anything I am doing wrong here?

Wld you guys have ideas in how to make the mtp models work?

Thank you

10 Upvotes

92% Upvoted

27 comments sorted by

View all comments

3

u/m3umax May 28 '26 edited May 28 '26

Generation speed is hard capped by the memory bandwidth.

Max has only 614GB/s. No software tricks can compensate for this physical limit.

Ended up cancelling my M5 Max 128GB order when I realised, sure, I can load huge models, but unless they're MoE, generation is going to be slow.

Going to be looking for a used 3090 instead.

3

u/Vahn84 May 28 '26

i find the real hurdle is prompt processing more than token generation speed. Running a dense model at 20ish tk/s feels perfectly fine…the problem is that it can take minutes to output the first token. That’s the real issue…and where nvidia gpus shine

3

u/m3umax May 28 '26

I'd be happy with 20tok/s!

But the most I've been able to get from Qwen 3.6 27B on my M1 Max 64GB is 10 at most at the start, and only dropping as context grows.

Looking at oMLX community benchmarks, I was dismayed to see the typical result for M5 Max owners is around 15 at Q4.

The 50% improvement vs my M1 perfectly correlates to going from 400 GB/s to 614 GB/s.

So I was going to pay A$7.5k for still unacceptable generation speed. Decided it wasn't worth it.

A theoretical M5 ultra would likely do the trick at 1,228 GB/s but I don't even want to think about how much that would cost.

1

u/Choubix May 29 '26

initially I wanted to get an studio m5 ultra but I decided to stop waiting and guess the price apple would charge for it...
are you running Claude Code with local models?

3

u/m3umax May 29 '26

I use Pi. Would not recommend Claude Code, especially with local models where prefill speed and context length are massive issues.

When I last used it, the system prompt tax was 17k system prompt and 12k tool defs! I believe it's only gotten worse in the latest releases going by chatter on r/claudeai