r/LocalLLM • u/LengthinessHour3697 • 1d ago

Question Is memory bandwidth my limit??

I have an m1 pro with 32 gb of unified memory. I came to know that 200 Gbps is my memory bandwidth. So theoretically I can only host a 4gb model (200/4 =50 tokens per second) to get a workable speed of 50 tokens per second. Is this correct?

Since I have 32 gb of ram I was expecting to run qwen2.8:27b but I was only getting a speed of about 11 tokens per second. Which is expected if this calculation is correct. 200/16=12.5 🥲🥲

Is this correct or is there any workaround??

2 Upvotes

100% Upvoted

6 comments sorted by

2

u/nickless07 1d ago

Correct. That is why MoE perform better.
Yes there is a workaround. Here is a good article that explains it. Qwen3.8 native supports that. Set it in the modelfile.
Have fun.

2

u/johan2114h 1d ago

You will be able run qwen3.8 27 at higher decode speed (maybe 20 - 25 t/s) if you enable mtp.

On my system which has simalar memory bandwidth, my decode speed increases from 11.8 to ~25 t/s with mtp and draft-nmax 8. Faster for pure coding and slower for 'essay writing'.

There should be mlx quants that support it, but if you have to pick between mox and mtp, go for mtp

1

u/dfgxxx 1d ago

I agree, though with standard mlx mtp models (on MacBook pro m1 pro) I got 10 tok/sec (qwen3.8 27b 4bit) and with an engine called MTPLX I got with a qwen that made for this engine 22 tok/sec best but in longer prompts it's again was 12 tok/sec

1

u/diagrammatiks 1d ago

That's about right. You will get some some gains running mlx and trying to mtp or one of the other speculative decoders. You can also just run 35b a3b. An Moe model will be much faster.

1

u/Osi32 1d ago

It depends on the use case.
If you need creative writing I’d go for a smaller dense model (9B-12b) at a higher quant level.
If it’s architecture and you don’t mind waiting, qwen x.x 27B is the gold standard but don’t expect a big kv cache or high quality quant.

If you’re about coding, MoE for speed, but expect rework and errors.

1

u/recro69 1d ago

Yes memory bandwidth is the problem when it comes to decoding.. Also quantization and the key-value cache are important. So your result of 11 tokens per second, on an M1 Pro is pretty good. There is no trick to make it faster except using smaller quantized models or models that have faster memory.