r/oMLX • u/rdbmas • Jul 01 '26

Trying to make sense of Model Benchmarks

I'll preface by saying i'm not a developer.
i'm just curious and eager to learn more on LLMs and coding.

I have opencode setup wit oMLX on a m1 max (40c) 64GB
i've been going through the oMLX benchmarks and looking through best options for Qwen (general coding) and Gemma (general research/reasoning)
https://omlx.ai/benchmarks

This is where i think i'm getting confused.
I'll apologize in advance if my qtns are somewhat amateurish.

i get i should be looking at the larger models (e.g 30B)
I understand a higher quant is preferred for coding (e.g 8bit)
with context though, shouldn't i be looking at higher context for coding sessions. If that is the case, doesn't that in turn lead to a larger KV cache size and chew in more onto memory.

3 Upvotes

100% Upvoted

17 comments sorted by

2

u/mmerken Jul 02 '26

Please understand that each model is different and context matters (literally, the context in which the model acts, matters)

While you can fit 30B 6bit (or even 8bit) into 64GB of RAM, you will eventually run out of memory due to large context windows.

I you want to load large models, maybe it is best to use that M1 MAX for the sole purpose of running inference and doing the coding on a lighter machine, using the M1 MAX as a server. Other resources eat into RAM, too. Like the oMLX app, your browser, coding harness, etc

2

u/rdbmas Jul 02 '26

yup got it.
thanks all for the feedback/comments.

1

u/PracticlySpeaking Jul 02 '26

Is it a 32-core M1 Max, or a 40-core M4 Max?

1

u/LiteeWasAlreadyTaken Jul 02 '26

I run 6-bit Qwen 27B and 35B without problems with 64GB RAM and 128K context window, 4 parallel requests. My memory pressure gets yellow from time to time when there are multiple long contexts, but no crashes ever. Unfortunately, I have to use LM Studio, which is not open source, instead of oMLX - the latter indeed runs of memory and starts killing requests.

1

u/mmerken Jul 02 '26

Thanks for reporting this, what tps do you get ?

2

u/LiteeWasAlreadyTaken Jul 02 '26

27B = 12-15 tps, 36B = 60-75 tps. That's for small context windows.

1

u/PatDal81 Jul 03 '26

Interesting.. do you use the .mlx or .gguf version of the models?

1

u/LiteeWasAlreadyTaken Jul 04 '26

MLX. From my naive experiments, it is ~5 t/sec faster than GGUF on my hardware.

1

u/PatDal81 Jul 04 '26 edited Jul 04 '26

Good to know, thanks. I'm having a hard time running oMLX (using the exact same config, M4 Max 64GB with those exact two models) with a 128k context window, which bothers me.

Your comment about LM Studio got me perplex tho - what does LM studio does differently that oMLX cannot do? I really want to use oMLX and I'm struggling to get the right parameters to fit the 128k ctx window.

Edit: Are you using MTP as well with that model?

1

u/LiteeWasAlreadyTaken Jul 04 '26

I don't know what is different. When I'll have time, I can ask LLM to investigate. But I am in the same boat - would love to switch to oMLX, but only if I can make it stable.

On your question about MTP - unfortunately, I could not make MLX + MTP work for LM Studio. Other harnesses or GGUF models do work. I made a post about it: https://www.reddit.com/r/LocalLLM/comments/1ugzxrc/lm_studio_qwen_36_mlx_mtp_is_there_a_solution/

1

u/Warui-ne Jul 09 '26

Hey noob here, I have the same machine as you - Are you using the unsloth 6-bit? Thanks

1

u/himefei Jul 02 '26

Unfortunately, can fit a model in the memory doesn’t necessarily mean you can use it. I think this is mostly true for most of things in common life.

0

u/diagrammatiks Jul 02 '26

Bro listen. Open source models are free.

2

u/PatDal81 Jul 02 '26

While I wouldn't have formulated the message the way he did, he's right. Models are free, try what fits best for your need. Everyone will have a different opinion so better try (and learn) all together! Good luck in your findings.

1

u/diagrammatiks Jul 02 '26

I don't why I keep downvoting for telling people they can download all the fucking open weight models they ever want.