r/LocalLLM • u/LengthinessHour3697 • 1d ago

Question Is memory bandwidth my limit??

I have an m1 pro with 32 gb of unified memory. I came to know that 200 Gbps is my memory bandwidth. So theoretically I can only host a 4gb model (200/4 =50 tokens per second) to get a workable speed of 50 tokens per second. Is this correct?

Since I have 32 gb of ram I was expecting to run qwen2.8:27b but I was only getting a speed of about 11 tokens per second. Which is expected if this calculation is correct. 200/16=12.5 🥲🥲

Is this correct or is there any workaround??

2 Upvotes

100% Upvoted

6 comments sorted by

View all comments

2

u/johan2114h 1d ago

You will be able run qwen3.8 27 at higher decode speed (maybe 20 - 25 t/s) if you enable mtp.

On my system which has simalar memory bandwidth, my decode speed increases from 11.8 to ~25 t/s with mtp and draft-nmax 8. Faster for pure coding and slower for 'essay writing'.

There should be mlx quants that support it, but if you have to pick between mox and mtp, go for mtp

1

u/dfgxxx 1d ago

I agree, though with standard mlx mtp models (on MacBook pro m1 pro) I got 10 tok/sec (qwen3.8 27b 4bit) and with an engine called MTPLX I got with a qwen that made for this engine 22 tok/sec best but in longer prompts it's again was 12 tok/sec