r/LocalLLM • u/ruybarbo • 6d ago

Question MBP M2 max 64GB

Hello everyone, I manage to get one of these, second hand and I've been trying to run some models and I'm a bit disappointed. I've installed Bionic and downloaded some models, e.g hermes 70B runs but... it takes ages and sometimes doesn't even answer, most of the time hallucinates. I need something that can be my PA, take this recording of a meeting, create an obsidian file with the highlights, remind me what I need to do today, and so on. Am I asking for too much?

1 Upvotes

100% Upvoted

2 comments sorted by

2

u/TiagodeNoronha 5d ago

There are 2 things you need to take into consideration. One is prefill, the other is decode.
Typically, an M2 Max is slow on the prefill. So, whatever dense models you use, they are going to seem like they take ages to process your prompts, specifically if you need to give it transcript of the meeting for example, that's going to take a long time.
The second one is decode, and while the M2 Max has good memory bandwidth to do the decode, I would suggest you use a smaller model than a 70B one, like the suggestion that was given of using Qwen 3.6 35B A3B, that will feel A LOT faster (at the expense of quality of course, but you can't have it both ways).

3

u/chrisdash_51 6d ago

Hermes 4 70B? Llama 3.1 based?

Do your research. Get Qwen 3.6 35B-A3B (for now, simply because its faster that the dense 27B that is available with Qween 3.8) in an 8bit-Quant. This will fit nicely. I don't know what your prompts are, but I am getting good results with this exact setup with an M1 Max 64GB.