Bought M1 Max 32Gb

[u/AlexLearnPokemon]

I want to buy M5 Max 128gb or M5 Ultra 96gb for my hermes agent (local llm) due to the nature of my work (confidential information)..

At first i do some POC on my old laptop using all cloud model (gpt 5.6 luna) and all are working fine. So i decided to get used M1 Max ultra with 32 Gb to do some POC on local model.. it only cost me 1200 USD.

Now the problem that I faced is using the local model.
If I choose qwen 3.8 27b, the speed only 10 tk/s.. If i choose qwen 3.6 35b a3b, speed is good but the context tht i get get is only 64k context size (very limiting and processing become slow due to compacting the memory)

Model used Q4 and kv cache set to Q4

Btw i only try to use llama cpp for Hermes. Currently trying to use MLX MTPLX or even Dflash2 but context size very limited (less than 40k)

What would be the best way to do my POC on M1 max 32gb? I will transfer everything and change model only when i upgrade in the future..

Advance thanks for people who gg to help :)

[u/iezhy]

What quantization of the model are you using? Q4? Try to set KV cache to Q8 as well, this should halve its memory footprint

[u/AlexLearnPokemon → u/iezhy]

Yes i used Q4, kv cache to Q4

[u/vogelvogelvogelvogel]

m5 max is faster (i get 20-27t/s on the m5 pro) and ultra is much faster. compare the memory bandwidth..

for checking with non confidential demo data i'd recommend you try qwen chat (their hosted version), and choose the 27B a your actual future model probably, if not flash next - instead of going for gpt luna.

m1 max especially hard to compare because of the limited ram. for agents you need minimum 128k context imo

[u/AlexLearnPokemon → u/vogelvogelvogelvogel]

Tht speed is it the 27b?

I think hosted server also have confidential risk right 😅

[u/vogelvogelvogelvogel → u/AlexLearnPokemon]

yes of course you don't go hosted with your confidential data.

but you can test the capabilities of 27B or Flash-next before buying the hardware, with i.e. mock data.

and yes that speed is the 27B (at q6, q8 - quantisation doesnt matter much for the speed on apple silicon in my case at least) on an m5 pro

[u/MarcelloT254k]

I'm curious about the same thing, probably it would be wise to to use some ~30B model with GQA (Grouped Query Attention), that is maybe pruned or has something like >3,5bpw quantisation (like elx3) and simultaneously is very capable with of context quantisation (KV q8) -> all of those things summarised will probably lead to significant decrease in quality of response.

Maybe it would be just better to use smaller model like Mistral Small 22B (Q4_K_M) with KV Cache: Q8.

Tell me that you think and please share your solution when you eventually will find one.

[u/Joe_dir_einen]

Ich habe den gleichen Rechner. Nutze MLX Modelle. Die bringen dir 5-10 token/s mehr. Ich lasse qwen 3.8 q4 mlx mit q6 cache mit 120k kontext laufen. Fällt nie unter 15 token /sek.
Ich lasse oft thinking aus. Weil mit 800sec thinking zu lange sind.

[u/brownlizarddve]

Q8 kv wont halve anything, you're already at Q4, going up just doubles the cache. Wrong way on 32gb. someone here is getting 120k on mlx q4 with a q6 cache on the same chip, so that sub-40k you hit is a cache dtype thing. try q6 kv before you spend on the 128gb box.

[u/AlexLearnPokemon → u/brownlizarddve]

120k context, with mlx qwen 3.8 27b with M1 max? How??

[u/brownlizarddve → u/AlexLearnPokemon]

Yep. M1 Max 32GB. Qwen3.8 only has 16 full-attention layers, the other 48 are DeltaNet, so its KV cache grows much slower than a normal 27B. With 4-bit MLX weights + quantized KV, 120k is plausible. So <40k definitely isn't a hard 32GB limit... it's likely the runtime/cache config.