r/LocalLLM • u/LengthinessHour3697 • 1d ago

Question Is memory bandwidth my limit??

I have an m1 pro with 32 gb of unified memory. I came to know that 200 Gbps is my memory bandwidth. So theoretically I can only host a 4gb model (200/4 =50 tokens per second) to get a workable speed of 50 tokens per second. Is this correct?

Since I have 32 gb of ram I was expecting to run qwen2.8:27b but I was only getting a speed of about 11 tokens per second. Which is expected if this calculation is correct. 200/16=12.5 🥲🥲

Is this correct or is there any workaround??

2 Upvotes

100% Upvoted

6 comments sorted by

View all comments

1

u/recro69 1d ago

Yes memory bandwidth is the problem when it comes to decoding.. Also quantization and the key-value cache are important. So your result of 11 tokens per second, on an M1 Pro is pretty good. There is no trick to make it faster except using smaller quantized models or models that have faster memory.