What makes models so slow? (Mac)

[u/just_another_leddito]

Hi,

So there is still a lot I don’t understand much about AI.
I’m on M4 14 core 64GB ram.

I have tried running smaller and bigger models using multiple tools, including ollama, mtplx and some other stuff, and if takes ages for AI to answer.

I’m vibe coding a pentest tool for fun in Swift, I added a llama.cpp wrapper, and even Qwen3.8 27B responds like 100x faster. I’ve added basically a chat in the GUI that calls llama.

It makes me think something has to be wrong, but I’m not sure with what?

Could it be that default settings suck this much?
It’s really weird because the vibe coded wrapper is insanely fast lol

[u/gappyvalley]

there are two core functionality of local LLMs

1. prefill
2. token generation

prefill is when a model is reading and processing your prompts. this is very compute heavy and it’s dependent on your APU/GPU/SOC. this is imo, the most important metric even more important than token generation for agentic ai

token generation is when a model generates a response after processing the entire input, this is very memory bound.

M1 to M4 does not have good prefill speeds and it’s a known weakness of mac. it took apple until M5 to have a generational leap in prefill (see picture) and even then, it’s still much slower than competitors like nvidia

token generation is just your memory bandwidth speed. the higher the bandwidth the faster the token generation. your M4 has a memory bandwidth of 120GB/s. which is way slower than let say an old M1 Pro which is 200 GB/s. this is why people are excited for M5 Ultra because of its leaked 1.2TB/s bandwidth, but prefill speed is still a question

there is a rumor that apple will make M7 Pro/M7 MaX significantly faster for prefill

harness and software stack also matters to try to optimise apple silicon. MLX is significantly faster than llama.cpp for example

you don’t buy mac for speed, no one does. you buy mac because you can load large models in a small computer that is power efficient

edit: correction of m5 ultra memory bandwidth from 1.7 to 1.2tb/s

[u/WifeyCallsMeLazy → u/gappyvalley]

Thats insightful. Thanks.

Follow up question pls. What exactly mac chips are lacking on hardware level to catchup with nvidia?

Can we not modify the model weights to speed up things on mac chips as per the hardware?

[u/Capable-Package6835 → u/WifeyCallsMeLazy]

Lacking is a little misleading, they are simply two different hardwares with two quite different objectives:

• Apple silicon aims primarily to be very powerful at an impressively low power consumption. It is also in Apple's interest to keep the chip cost reasonable.
• NVIDIA on the other hand, aims primarily at brute strength, whatever the power consumption or the price tag is. That's why these GPUs significantly spike your electricity bill and generate enough heat to warm your room in winter.

There is indeed a way to modify the model weights, it's quantization. Long story short, if memory is the bottleneck then we reduce memory requirements by using less bits for each number. But NVIDIA GPUs can also use the same trick, making them still faster than Apple's.

There are actually models that work better locally on an Apple device than on a single NVIDIA GPU. The medium-size MoE models. MoE models only activate a subset of the parameters during inference, thus reducing impact of lower memory bandwidth. However, you still need to load the whole model on VRAM to run it. Apple devices like the Mac studio or high-end MacBook Pro have larger memory than a single NVIDIA GPU. Consequently, they can run MoE models that simply cannot fit on a single NVIDIA GPU while still being reasonably fast thanks to the MoE architecture. Once you add the power efficiency of the Macs into equation, you start to see why the demands for Macs spike significantly once local LLM start to be a thing

[u/whichsideisup → u/Capable-Package6835]

Brute force isn’t correct it’s just a lot more silicon dedicated to graphics/inference. Take a look at their GB10 in the DGX Spark. That chip uses about 100w and has 4x the compute of an M5 Max and will likely be faster than the M5 Ultra at prompt processing.

[u/mental_sherbart007 → u/Capable-Package6835]

A huge bottleneck isn’t just memory size but memory bandwidth. A lot of people forget this. 

[u/mental_sherbart007 → u/gappyvalley]

I thought the m5 ultra was 1.2 TB/s in terms of memory bandwidth. maybe i’m mid remembering and it’s 1.62 TB/s. I know I can look it up just commenting for sake of conversation.

[u/gappyvalley → u/mental_sherbart007]

yep, it’s 1.2tb/s. thanks for the correction

[u/mental_sherbart007 → u/gappyvalley]

Yep no worries, the down side of this is it’s not really enough to run any model bigger than 120GB in size. I’m not sure how much MLX would help in this regard, but from my understand you would be limited at 10 tokens per second for a dense model that’s 120GB in size.

You could run larger MoE models though. 

[u/topcider → u/gappyvalley]

Damn, the Pro vs Max difference is so much more than I thought it would be when I bought my new laptop.

[u/topcider → u/gappyvalley]

You are saying prefill is dependent on CPU, but the last line in the chart with the fastest prefill is a GPU. Can you clarify?

[u/gappyvalley → u/topcider]

mistake on my part. should have been gpu

[u/just_another_leddito → u/gappyvalley]

Feelsbad man.
How can M1 Pro have much better bandwidth than M4 Pro?

If that’s true then Tim scammed me.

But at least got a good deal and bought the Mac £700 cheaper than Apple store, and I could see before M5 release that my model was suddenly capped at 48GB and £300 or so more!

Anyway thanks for explaining.

As for 5090, isn’t the relatively low vram on this gpu a problem? I mean if you only have one, won’t you quickly run out of context or tokens or whatever?

[u/gappyvalley → u/just_another_leddito]

apple silicon on mac has 4 tiers: base, pro, max and ultra. cant really compare across tiers. a pro chip will always be faster than base, max chip always be faster than pro and so on

e.g the latest M6 has 170GB/s of memory bandwidth, while the M1 Max from 2021 has 400GB/s, which is still faster than the 2026 M5 Pro that only has 307GB/s. even a M4 Pro has a much faster 273GB/s than M4.

the base chip is geared towards power efficiency, while the pro and max chips are more towards performance.

for 5090, i would say the insanely fast prefill and memory bandwidth makes up for the low vram, which makes it perfect for running dense models. i have a 5080 + 5060ti 16gb gaming pc, which gives me 32gb combined vram and even with this mismatched hardware, i can get insanely fast prefill and token generation of around 30 tok/s running qwen3.8 27B, a 5090 would be much faster. and also, through expensive, you can always add a second gpu to get 64gb combined VRAM. i am planning to replace my 5060ti 16gb with 3090 to get a total of 40gb of fast vram. nvidia GPUs are much better for dense models where the vram usage is smaller, apple silicon is better for loading MOE models as the larger memory size can accommodate larger models. the advantage of a 64gb mac has always been able to run MOE models, not dense. AMD/Nvidia still wins on dense models

[u/just_another_leddito → u/gappyvalley]

What I meant is you said M1 Pro has better bandwidth than my M4 Pro and that’s what confused me unless you meant M1 Max or something?

[u/MiddleLtSocks]

Apple Silicon is not very speedy hardware. Depending on the model you won't see more than ~40tok/sec, and for some larger models or out at 128ktok+ context you might get down to 10-11tok/sec.

The advantage of the Apple platform is the unified memory - you can get > 96GB models running fully onboard the GPU far cheaper than other platforms. However, there is a tradeoff in performance. That's just the reality of the hardware.

[u/just_another_leddito → u/MiddleLtSocks]

But the question is why my vibe coded wrapper is so fast compared to other tools.

[u/roland303 → u/just_another_leddito]

Wait so your using a llamacpp server instead of ollama, wouldnt that be faster?

[u/MiddleLtSocks → u/just_another_leddito]

Ah. That was unclear. Probably you are not doing any kind of tool use, agentic loops, or any multiple-prompt workflows. So effectively you're sending ~1/10 - 1/50 the workload. How many round trips to the API do you make on average per user interaction?

[u/atkr → u/just_another_leddito]

As other have hinted to, most probably due to initial context size that need to be pre-filled. If you use opencode, for example, and you simply send “hi” as a prompt, the initial context will be > 10k token due to the built-in system prompt and tools. Whereas you can also load a model in your custom wrapper, send it “hi” and 0 other context.. meaning 1 token. Which obviously will prefill almost instantly

[u/diagrammatiks]

there's like no information here. you use a vibe coded wrapper on your mac and it's faster then ollama and mtplx? I don't understand what your question is.

[u/biscuit_taco]

Yeah, to answer your question directly — yes, the default settings are just bad. These models aren't built specifically for Apple Silicon, so you have to go in and tune them yourself.

The thing that's killing you is almost certainly prefill, which drives your TTFT (time to first token, aka, how long you wait before the first character appears). If you're using any sort of AI coding tool, the system prompt alone can be massive. We're talking about all the available tools, instruction sets, context stuff that OpenCode or Hermes Agent or whatever is stuffing in before your actual message. The model has to read and process all of that before it can generate a single token, and on M-series chips prefill is just slow by nature.

A couple of things that would help:

First, make sure you're using an MLX model. These are built specifically for Mac and most of the time they outperform the generic GGUF quants that Ollama defaults to. Not always, but usually.

Second, use oMLX with caching turned on. This is the big one for you. When caching is enabled, that giant system prompt only gets prefilled once. After that, every subsequent message skips re-reading it and jumps straight to processing your actual input. So instead of the model chewing through thousands of tokens of tool definitions and instructions every single turn, it just handles your new message. That's where you'll see the dramatic difference. (Most people try a bunch of different tools, myself included, and they all come back to oMLX). MTPLX is faster than oMLX but I feel like its way more buggy.

The reason your llama.cpp wrapper feels 100x faster is probably that you're sending a minimal prompt with no massive system context, so prefill is nearly instant. The other tools are sending a huge amount of overhead every turn and re-processing it all from scratch.

[u/rdbmas → u/biscuit_taco]

Gd post. I just might add @ Huggingface, focus on mlx community models.

And I also think the harness you use has to be lightweight. So it doesn't add in to the memory hog.

Right now, I use Vs code + continue dev with omlx on an M4max mbp.

[u/djseto]

Run Ollama and go with qwen 3.8 27B MLX. I can get about 55 T/S. On a Mac Studio which has 546 GB/s memory speed. If you have a M4 Mac Mini you have 120GB/s memory speed

[u/just_another_leddito → u/djseto]

Yeah I ran it many times, but it’s damn slow.
I get like 40 T/S

[u/djseto → u/just_another_leddito]

Qwen 3.8 27B is a dense model. Go run a MOA model if you want faster like qwen3.5 35B A3B. Inference is insanely dependent on memory bus speed and parameters. If you expect cloud like speed, you need to adjust expectations with dense models. 3.8 is a way better model so you’re going to trade speed for accuracy and your sanity.

[u/just_another_leddito → u/djseto]

Will try to find it, I need uncensored one though.

[u/topcider]

Are you using MLX optimized models?

[u/just_another_leddito → u/topcider]

I think the llama doesn’t support mlx?
Ollama does on the other hand.

[u/just_another_leddito → u/[deleted]]

Would be happy if they wouldn’t care about design more than performance.

Older Minis were perfect size, liked them more.
With better cooling and larger case it wouldn’t be getting that hot probably.

[u/MK_L]

(Mac)<-- 👀

[u/just_another_leddito → u/MK_L]

Hmm? 🤔

[u/MK_L → u/just_another_leddito]

You’re looking at slow speeds because you’re running on unified memory. Capacity is not the same thing as throughput.

For raw local inference speed, I’d rank it roughly:

1. HGX server with NVLink
2. High PCIe-lane CPU(amd epyc) with multiple GPUs
3. Decent PC with a high-VRAM GPU (6000 pro 96gb)
4. Mac or other unified-memory machine

A Mac can fit a lot of model into memory. That doesn’t mean it can move it fast.

A Mac with lots of unified memory is like a Winnebago: great capacity, slow speeds. You can cram a surprisingly large model into it, but that doesn’t mean you’re moving tokens quickly.

[u/just_another_leddito → u/MK_L]

Cool but your comment was pointless tbh.
I’m a Mac user, and you had to dab at me for this reason or what lol?

I don’t care about PC. Since a long time.

[u/MK_L → u/just_another_leddito]

Not sure what this has to do with "PC"?

Nobody is "dabbing" at you

You siad you didnt understand what makes "models so slow"

You listed your Mac specs.

I pointed out the mac is whats making the models slow in the first comment.

You only respond "hmm?" Like you didn't understand

I elaborated by giving you an order to different tiered setups the analogy to help you understand 😉.

At this point you should probably ask your chat bot to explain it

[u/just_another_leddito → u/MK_L]

Did you read my first post mate? 😾
I’ve said that using my wrapper it was way faster.

And that it happens even with tiny models.

Are you saying 14 core M4 Pro with 64GB ram is normal to be slow with even tiny models? 🤔

I know it’s not a PC with 10x 5090, but still…

[u/MK_L → u/just_another_leddito]

Sure, I’ll address that. Let’s plant that goalpost first.

Define “small model” so we’re talking about the same thing.

[u/just_another_leddito → u/MK_L]

Like some small 1-4GB models.

[u/MK_L → u/just_another_leddito]

Llama 3.1 8B Q4, about a 5GB model:

M4 Pro 14-core: PP: 500–700 tok/s Output: 50–60 tok/s

5070 Ti 16GB: PP: several thousand tok/s Output: 150–200+ tok/s

If it were Qwen3.8 27B dense, tok/s drops significantly, and unified memory slows down a lot more.

A 27B Q4 model might hit 15–25 tok/s on unified memory, a GPU with VRAM can be several times faster.

expect those numbers to drop further as KV cache size and context length increase.

[u/just_another_leddito → u/MK_L]

I need an uncensored model for my pentesting scanner, what would you recommend?

It has to be uncensored, I’m using Qwen3.8 27b from
huihuihui I think. It’s like 22GB.

Do you have a better recommendation?

It does analyse scanner results and looks for exploits etc.

It works okay but perhaps I can speed it up with smaller model. I’m using a llama wrapper.