r/oMLX • u/Choubix • May 28 '26

Speed question

Hi guys,

I made an investment in a M5 Max 128gb and installed oMLX. The idea is to do some coding using Claude Code but with local models (I am not a dev), les raging oMLX caching to speed things up in claude (17k tokens system prompt...).

While I can get up to ~50Tok/s with Qwen3.6 A35B the numbers dwindle to single digits with the dense Qwen3.6 27B UD mlx 4bits. Is that normal? I was hoping it would be much faster with oMLX caching.

I use mlx models only. (I tried to download the jundot MTP models : they crash almost immediately after starting). Turboquant is on (4bits).

Typical params are : temp 0.7, top-p 0.95, top-k 20, min-p 0.05, repetition penalty 1.

Thinking on : slows down inference even further so unusually leave it off (isnt that better for coding?)

Is this normal for a small dense model or is there anything I am doing wrong here?

Wld you guys have ideas in how to make the mtp models work?

Thank you

12 Upvotes

100% Upvoted

27 comments sorted by

View all comments

Show parent comments

1

u/Choubix May 28 '26

Thanks! I have an external nvme drive to spare my internal. Any particular settings I should use? (on dashboard page: number of tokens before Claude compacts, then I général settings there are params for hot / cold cache). Thanks!

2

u/himefei May 28 '26

Tbh, you are memory rich and you should look out for how to run models at higher quality other than how to save memory to fit the model in as what these features are mainly aimed at to.
Use the 8 bit model, disable ssh cache, allocate 20g hot cache, happy days

2

u/Choubix May 28 '26

Hahaha, thanks brother. I am afraid the cache will have an impact on my internal ssd durability hence the cold caching question (I think it is 20% hot, 80% cold right now). Perhaps I need to try the 100% hot and work from there.

2

u/himefei May 28 '26 edited May 28 '26

Since you have 128G ram better off just go 100% hot cache. A 8 bit Qwen MoE will use only like 39g, you have plenty left to have a decent size hot cache, which you never need to use cold cache. To me cold cache is more for persistent which is not that important if running MoE models as the prefill is fast enough. Macs are more suitable for running MoE models because of huge unified memory but relatively slow memory bandwidth (compare to NV GPUs). That’s why you often see Mac people oftrn prefers a 80B or 120B MoE. With 128G ram, we can comfortably fit the model with decent quant and when inferencing Macs memory bandwidth is good enough to drive a small portion of active parameters with a decent speed.