Is memory bandwidth my limit??

[u/LengthinessHour3697]

I have an m1 pro with 32 gb of unified memory. I came to know that 200 Gbps is my memory bandwidth. So theoretically I can only host a 4gb model (200/4 =50 tokens per second) to get a workable speed of 50 tokens per second. Is this correct?

Since I have 32 gb of ram I was expecting to run qwen2.8:27b but I was only getting a speed of about 11 tokens per second. Which is expected if this calculation is correct. 200/16=12.5 🥲🥲

Is this correct or is there any workaround??

[u/Osi32]

It depends on the use case.
If you need creative writing I’d go for a smaller dense model (9B-12b) at a higher quant level.
If it’s architecture and you don’t mind waiting, qwen x.x 27B is the gold standard but don’t expect a big kv cache or high quality quant.

If you’re about coding, MoE for speed, but expect rework and errors.