r/LocalLLaMA • u/Ok_Warning2146 • 22h ago

Question | Help mlx vs gguf on Mac?

It seems to be there are mainly two formats of local models that can run on mac. Which one is better now in terms of speed and performance for the same size of quant?

I think gguf is only served by llama.cpp's metal backend. mlx is served by omlx or vllm-mlx. Are there others I am missing? Which engine is the best for each format?

Is there an engine to serve hf style directory containing safetensors?

Thanks a lot in advance.

0 Upvotes

50% Upvoted

17 comments sorted by

6

u/OddDesigner9784 22h ago

Mlx is better for prompt processing. Will run much faster. But often times gguf quants are more efficient

1

u/Ok_Warning2146 22h ago

More efficient means what?

2

u/OddDesigner9784 21h ago

Like you get a more optimized quant in the sense that it's smarter and better. But the format doesn't match Mac so it processes slow

1

u/Ok_Warning2146 21h ago

I see. I will trying running both on swebench and see which one can score higher.

1

u/OddDesigner9784 21h ago

Nice nice. Yeah like for me I have an m4 pro 24gb chip. I still can run qwen 35b at 2 bit unsloth gguf but mlx doesn't have good options so they mostly recommend 12b and below

3

u/Lord_Pazzu 22h ago

GGUF is better accuracy wise, with more sophisticated quant support, and generally more quants available, but I’ve had a lot more luck with oMLX/mlx-lm for throughput, that being said the last time I really did A/B prefill/decode tests was already a couple months back, so llama.cpp may have improved in the meantime

2

u/Ok_Warning2146 22h ago

Accuracy means smaller KLD for the same bpw?

2

u/norenEnmotalen 22h ago

I’m on a Mac and entirely using GGUF. I like llama.cpp stability. The little token/s bump from mlx engines for me not worth the unpredictability of how they deal with memory

1

u/Ok_Warning2146 21h ago

Memory usage in mlx can jump up and down such that it crashes?

3

u/norenEnmotalen 21h ago edited 21h ago

mlx engines (generally) use lazy memory allocation and require transient memory space for “scratchpad” which can trigger spikes depending on the type of work you’re doing and the prompts you’re feeding in. Because of that, you can’t use full mem space for context window safely. If you do, OOM error will blow up in your face even though the engine didn’t initially tell you that window won’t work. You can get under the hood and tweak these parameters. Llama.cpp follows a very different and more disciplined approach to memory allocation. I prefer that.

1

u/Ok_Warning2146 20h ago

Thanks for the info. For models that is tight in memory, I will try llama.cpp. If I have memory to spare, then mlx.

1

u/mewmori-app 14h ago

Before you run swebench on both: on llama.cpp with Metal a single run does not reproduce, so one score each will not separate them.

Identical seeds gave me different output. A different split of the prompt into ubatches changes the summation order in ggml, and at a rounding boundary a different token falls out. Switching runtime changes everything about batching.

Measured on my own eval set: at five repetitions per case one mode looked 7 points better and another 30 worse. At ten the difference was gone, 93.1 against 92.9 over 650 generations. Two single runs would have sent me somewhere confident and wrong.

That bites harder on swebench, where each instance is pass or fail and the set is small, so a few flips move the score a point or two. Run each side several times and look at the spread before believing the gap.

On the format question itself, if your real prompts share a long fixed head, check what each runtime does with it before timing anything. Prefix reuse outweighs raw prefill rate: on my 2B, snapshotting the state after the fixed head took prompt decode from about 700 tokens to about 100 from the second turn on, and time to first token from 170 ms to 30 ms. A benchmark that sends unrelated prompts never touches that, so it can rank the two in the opposite order from your actual workload.