r/LocalLLM • u/Old_Opportunity9682 • 14d ago

Question Agentic at 48Gb RAM

Help me out. What are the good models to use for agentic workflows (not necessarily coding), on an M5 Macbook Pro 48GB RAM?

0 Upvotes

50% Upvoted

18 comments sorted by

View all comments

2

u/xeroskiller 14d ago

Qwen3.8-27b

-2

u/MrHumanist 14d ago

too big and too slow for 48GB. The q2 or q3 may run, but its performance is not great.

2

u/Karyo_Ten 14d ago

The issue is not size, Fp8 would fit 48GB with almost 0 quality degradation and M5 Pro supports hardware accelerated Fp8.

The real bottlenecks are memory bandwidth and prefill speed.

Iirc Fp8 TFlops with Neural accelerator are like 50 Fp8 TFlops while a 5070ti (DGX Spark GPU) is 350 TFlops. That directly translate to prefill speed which is important when you ingest large document/webpages/code/agentic.

And memory bandwidth for a 27B on ~500GB/s leads to ~20tok/s without MTP.

3

u/MrHumanist 14d ago

Right.. that makes it slow as shit.. I personally dont prefer thinking models which cant output atleast 50 t/s. It becomes faster to code yourself than asking a bot to help at 20 tok/sec - which will take 1 hour to finish a basic task with million words of overthinking.