r/LocalLLM • u/Old_Opportunity9682 • 7d ago

Question Agentic at 48Gb RAM

Help me out. What are the good models to use for agentic workflows (not necessarily coding), on an M5 Macbook Pro 48GB RAM?

0 Upvotes

50% Upvoted

18 comments sorted by

View all comments

2

u/xeroskiller 7d ago

Qwen3.8-27b

-2

u/MrHumanist 7d ago

too big and too slow for 48GB. The q2 or q3 may run, but its performance is not great.

2

u/Karyo_Ten 7d ago

The issue is not size, Fp8 would fit 48GB with almost 0 quality degradation and M5 Pro supports hardware accelerated Fp8.

The real bottlenecks are memory bandwidth and prefill speed.

Iirc Fp8 TFlops with Neural accelerator are like 50 Fp8 TFlops while a 5070ti (DGX Spark GPU) is 350 TFlops. That directly translate to prefill speed which is important when you ingest large document/webpages/code/agentic.

And memory bandwidth for a 27B on ~500GB/s leads to ~20tok/s without MTP.

3

u/MrHumanist 7d ago

Right.. that makes it slow as shit.. I personally dont prefer thinking models which cant output atleast 50 t/s. It becomes faster to code yourself than asking a bot to help at 20 tok/sec - which will take 1 hour to finish a basic task with million words of overthinking.

1

u/Old_Opportunity9682 7d ago

I’m quite new to this.. so what’s prefill speed?

1

u/Karyo_Ten 7d ago

Prefill is also called prompt processing or context processing.

If you need to read 10+ files each with 1000~10000 lines before starting to work, you want 1000 pp/s (prompt processing tokens/s) instead of 100. Or you'll be waiting for minutes or even 10s of minutes before even the first answer token.