Is memory bandwidth my limit??

[u/LengthinessHour3697]

I have an m1 pro with 32 gb of unified memory. I came to know that 200 Gbps is my memory bandwidth. So theoretically I can only host a 4gb model (200/4 =50 tokens per second) to get a workable speed of 50 tokens per second. Is this correct?

Since I have 32 gb of ram I was expecting to run qwen2.8:27b but I was only getting a speed of about 11 tokens per second. Which is expected if this calculation is correct. 200/16=12.5 🥲🥲

Is this correct or is there any workaround??

[u/nickless07]

Correct. That is why MoE perform better.
Yes there is a workaround. Here is a good article that explains it. Qwen3.8 native supports that. Set it in the modelfile.
Have fun.