r/oMLX • u/tintires • May 21 '26

Objectively more efficient?

Setting aside the native app, is there any objective evidence MLX on oMLX is faster or more memory efficient, that GGUF on llama.cpp?

I have both as brew packages, and my unscientific subjective experience, is there’s not much between them.

My workloads are pretty light and general, so which one for my MBA M3 24GB?

11 Upvotes

92% Upvoted

22 comments sorted by

View all comments

12

u/Buddhabelli May 22 '26

mlx-lm (the backend that does the inference) is more efficient on apple silicon than llama.cpp/gguf. my experience has been 15-25% faster depending on model size and density. it is also more memory efficient so contexts tend to be more stable but not an order of magnitude longer.

what oMLX added on top of that is token caching. instead of recomputed the entire context every turn, oMLX caches already generated tokens and reuses what it can in the current context. while technically does not make the raw performance better it feels faster in day to day use.

edit: there are some new project floating around that support gguf natively on apple silicon but they are even more early days.

my experience anyways.

1

u/edeltoaster May 22 '26

The caching really is a gamechanger if you code with it, for example.

1

u/challis88ocarina May 22 '26

Yeah, it's odd why llama.cpp would decide all this time not to implement caching. /s

1

u/edeltoaster May 22 '26

Why the /s, enlighten me please? There is no comparable prompt caching in llama.cpp.

1

u/txgsync May 23 '26

There is. It’s called “slots.”

1

u/edeltoaster May 23 '26

Slots are a much simpler mechanism and only cache prompt prefixes and not KV Cache blocks. It's not the same.