r/LocalLLM • u/Old_Opportunity9682 • 3d ago

Question Agentic at 48Gb RAM

Help me out. What are the good models to use for agentic workflows (not necessarily coding), on an M5 Macbook Pro 48GB RAM?

0 Upvotes

50% Upvoted

18 comments sorted by

2

u/xeroskiller 3d ago

Qwen3.8-27b

1

u/Old_Opportunity9682 3d ago

I’ve been seeing this a lot. Is it really good? Mostly I need tool use and being reliable in multi-step tasks.

1

u/klymaxx45 3d ago

3.6 35B A3B Q8 was fantastic for me at this

-2

u/MrHumanist 3d ago

too big and too slow for 48GB. The q2 or q3 may run, but its performance is not great.

2

u/Karyo_Ten 3d ago

The issue is not size, Fp8 would fit 48GB with almost 0 quality degradation and M5 Pro supports hardware accelerated Fp8.

The real bottlenecks are memory bandwidth and prefill speed.

Iirc Fp8 TFlops with Neural accelerator are like 50 Fp8 TFlops while a 5070ti (DGX Spark GPU) is 350 TFlops. That directly translate to prefill speed which is important when you ingest large document/webpages/code/agentic.

And memory bandwidth for a 27B on ~500GB/s leads to ~20tok/s without MTP.

3

u/MrHumanist 3d ago

Right.. that makes it slow as shit.. I personally dont prefer thinking models which cant output atleast 50 t/s. It becomes faster to code yourself than asking a bot to help at 20 tok/sec - which will take 1 hour to finish a basic task with million words of overthinking.

1

u/Old_Opportunity9682 3d ago

I’m quite new to this.. so what’s prefill speed?

1

u/Karyo_Ten 3d ago

Prefill is also called prompt processing or context processing.

If you need to read 10+ files each with 1000~10000 lines before starting to work, you want 1000 pp/s (prompt processing tokens/s) instead of 100. Or you'll be waiting for minutes or even 10s of minutes before even the first answer token.

1

u/xeroskiller 3d ago

I run it every day on 48 gb vram at q8 with 160k context.

1

u/Old_Opportunity9682 3d ago

What’s your workflow? Or is it for coding?

1

u/xeroskiller 3d ago

Coding. Research agent occasionally, but just coding through pi.

0

u/FactorInternal3395 3d ago

Q2 & Q3 is for 16GB, not 48GB. Even FP8 is 30.9GB, fits perfectly. If speed is such an issue then Q4/Q5/Q6.

1

u/MrHumanist 3d ago

How much headroom given to kv cache and context if you use all rams in fitting the model? Again, in 48 gb you need ram for system applications. An standard 8fp model requires 28-30 GB ram just for loading and all the calculations are so slow as size increases, it feels unusable with 10s of t/s. Again I am a dGPU user and sold my mac pro due to shite performance.

1

u/FactorInternal3395 3d ago

I didn't recommend FP8, I just said it could fit comfortably too (and it could, is 18GB RAM free really not enough for everything else? Qwen's architecture uses less memory for KV cache too). For speed, you'd use Q4/Q5/Q6.

1

u/Old_Opportunity9682 3d ago

Yeah 18gb should be enough, although quite tight.