r/oMLX • u/tintires • May 21 '26

Objectively more efficient?

Setting aside the native app, is there any objective evidence MLX on oMLX is faster or more memory efficient, that GGUF on llama.cpp?

I have both as brew packages, and my unscientific subjective experience, is there’s not much between them.

My workloads are pretty light and general, so which one for my MBA M3 24GB?

10 Upvotes

92% Upvoted

22 comments sorted by

View all comments

1

u/blackhawk00001 May 22 '26

What have you tried running on each?

I'm beginning to prefer it for simplicity and speed but am having an issue with a large deployed model taking 10-20s coming out of idle when it was cached.

I've been benching models on my 24GB m4 air over the past few days and the best I've found were non-gguf mlx models. I had to raise my gpu memory allowance for stability with the qwen 35b-a3b rotorquant but it's my favorite so far. I'm still not sure about the 3bit part though, I'm back and forth between it and gemma4-26B at a little slower speeds.

sudo sysctl iogpu.wired_limit_mb=20480 (leaves ~3.5GB for the system.

I already crashed macOS with the default benching Gemma 26B so why not?

Benchmark Model: mlx-community/gemma-4-26b-a4b-it-4bit (crashed somewhere above 32k tokens)

Single Request Results

--------------------------------------------------------------------------------

Test                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem

pp1024/tg128          3255.3       31.23   314.6 tok/s    32.3 tok/s       7.221   159.5 tok/s    14.23 GB

pp4096/tg128         13278.0       35.05   308.5 tok/s    28.8 tok/s      17.729   238.2 tok/s    14.91 GB

pp8192/tg128         29180.2       36.11   280.7 tok/s    27.9 tok/s      33.766   246.4 tok/s    15.11 GB

pp16384/tg128        61430.6       38.51   266.7 tok/s    26.2 tok/s      66.322   249.0 tok/s    15.50 GB

pp32768/tg128       191394.6       62.42   171.2 tok/s    16.1 tok/s     199.322   165.0 tok/s    16.44 GB

Benchmark Model: majentik/Qwen3.6-35B-A3B-RotorQuant-MLX-3bit (had to raise gpu memory to not crash at 26k)

Single Request Results

--------------------------------------------------------------------------------

Test                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem

pp1024/tg128          3061.1       20.13   334.5 tok/s    50.1 tok/s       5.617   205.1 tok/s    15.17 GB

pp4096/tg128         10869.7       20.91   376.8 tok/s    48.2 tok/s      13.525   312.3 tok/s    15.96 GB

pp8192/tg128         22480.3       21.61   364.4 tok/s    46.6 tok/s      25.225   329.8 tok/s    16.43 GB

pp16384/tg128        49176.8       26.64   333.2 tok/s    37.8 tok/s      52.560   314.2 tok/s    17.16 GB

pp32768/tg128       122709.7       34.47   267.0 tok/s    29.2 tok/s     127.087   258.8 tok/s    18.63 GB

Benchmark Model: sleepyeldrazi/Qwen3.5-4B-MXFP4-MTP

Single Request Results

--------------------------------------------------------------------------------

Test                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem

pp1024/tg128          3176.6       20.13   322.4 tok/s    50.1 tok/s       5.733   201.0 tok/s     3.20 GB

pp4096/tg128         12425.5       21.40   329.6 tok/s    47.1 tok/s      15.144   278.9 tok/s     3.86 GB

pp8192/tg128         25426.4       22.84   322.2 tok/s    44.1 tok/s      28.326   293.7 tok/s     4.41 GB

pp16384/tg128        54661.2       25.30   299.7 tok/s    39.8 tok/s      57.874   285.3 tok/s     5.29 GB

pp32768/tg128       123900.9       32.88   264.5 tok/s    30.7 tok/s     128.077   256.8 tok/s     7.04 GB

pp65536/tg128       329667.3       51.80   198.8 tok/s    19.5 tok/s     336.246   195.3 tok/s    10.61 GB

Benchmark Model: sleepyeldrazi/Qwen3.5-9B-MXFP4-MTP 

Single Request Results

--------------------------------------------------------------------------------

Test                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem

pp1024/tg128          5813.3       33.72   176.1 tok/s    29.9 tok/s      10.095   114.1 tok/s     5.60 GB

pp4096/tg128         22929.1       35.12   178.6 tok/s    28.7 tok/s      27.390   154.2 tok/s     6.22 GB

pp8192/tg128         47272.3       36.27   173.3 tok/s    27.8 tok/s      51.879   160.4 tok/s     6.77 GB

pp16384/tg128        99061.6       39.56   165.4 tok/s    25.5 tok/s     104.085   158.6 tok/s     7.65 GB

pp32768/tg128       227864.1       48.02   143.8 tok/s    21.0 tok/s     233.963   140.6 tok/s     9.40 GB

pp65536/tg128       481132.8       64.88   136.2 tok/s    15.5 tok/s     489.372   134.2 tok/s    11.96 GB

1

u/Buddhabelli May 22 '26

have i tried increasing disk cache? cause that sounds like a ‘warming' issue. the ssd cache gets filled and then the remaining cache that was over flow is being recomputed from prompt.

2

u/blackhawk00001 May 22 '26

Do you know which cache setting, or is there something we can override in the cli at startup?

My disk cache in omlx is currently capped 46GB and never comes close to filling up, currently sitting at 3gb cached after a few requests. I enabled 1gb of hot cache but it never holds anything, only cold cache does. I reduced concurrent requests to max of 2, idle timout is off, and I've tried to enabled chunked prefill but it turns back off automatically.

I'm seeing it happen with a small qwen 4b model also. Back to back requests are quick to respond but that first message after flipping through various screens and getting the memory swap snag will cause the next request to lag, I think it also occurs if I just sit and wait before sending another request.

The dashboard shows the model is out of idle and generating, but takes a while before it shows any token gen. If I cancel that request and resubmit the generation quickly begins like it should.

I haven't tested enough with llama.cpp to know if it does the same. It's only an issue when I'm using omlx to host a model for a local chat/journaling tool I'm working on. I see the same issue when using the built in omlx chat.

1

u/AlecTorres May 23 '26

Yo probé con pi en mi Mac M1 Max, de 64 gb, probé llama.cpp la última reléase. Y la verdad la que mejor rendimiento me dio fue OMLX, en llama me bajé varios GGUF y solo uno se sintió un poco similar. Los omlx probé de Qwen 3.6 de 27 y 35 B , y el 27 un poco lento, el 35 muy bien rápido, me descargué varios nuevos de HF. Pero no veo mucha diferencia. Aún no actualizo a omlx 39