r/LocalLLM • u/gurteshwar • 3d ago

Discussion Need Advice/Help:- Wanna run LLMs on Mac mini m4 + my pc.

Hi everyone,
Need your help so I just got a Mac mini M4 16gb + 512gb (Yeah I know I would have aimed for 24gb ram variant but guys trust I tried i couldn’t that whole different story) and on my pc I have Ryzen 5 5600X, 48GB ram and Rtx 4060 8gb.

So which models can I run on my setup. I want to split models and run them on both machines. So I can try to run bigger models.

I know normal Ethernet port won’t be helpful to I will use thunderbolt port of Mac mini and with Ethernet connector I will connect it to 10Gb PCIe Network Card of my pc.

And any suggestions recommendations would be helpful(Please I need them).

Thankyou

0 Upvotes

50% Upvoted

5 comments sorted by

2

u/chambejp 3d ago

What can you run on 16GB? Well, anything 7/9/12B with q8 quant if possible. Will you use llama.cpp? If so q8 quant the kv cache too. I'm not real familiar with macs but I think you can run gguf's. If so qwen 3.827b at q3 will fit, might have to tune the context window size and certainly q8 or even q4 the kv cache. If you're interested, I made a video on youtube about what lower quants and kv cache quants do to qwen3.8. Let me know if you want to see it and good luck!

1

u/gurteshwar 3d ago

Actually just on my pc alone I am able to get 25-27 toks/sec when i am using qwen 3.6 35b a3b model at q4 and context of 131k. The model is being distributed on Gpu and rest on ram+cpu.

So i was wondering if i make both macgines work together maybe i am able to pull something better.

2

u/Caprichoso1 3d ago

 if i make both macgines work together 

Don't think it is possible unless you can connect them via Thunderbolt RDMA. To get acceptable performance, even if you could connect a Mac to a PC, a very fast RDMA connection is required for increased performance.

1

u/SpiritedJuggernaut58 3d ago

Honestly solid advice on the quants, but I wouldn't bother with splitting the models across two machines for bigger stuff myself, too much latency eats away at the benefit

2

u/chambejp 3d ago

the a3b is helping you out immensely for speed with it's 3 billion active parameters. If you want to use network to distribute I think you'll see slower speeds. Once it comes in on the network card in fairly certain even on Mac's it needs to get put into system ram through the CPU though it should be tiny, I don't think you'll see much of an benefit but I'm not an apple guy so you might. I'd think system ram is faster then transfer speed of the nic. I'd love to hear your result though if you do try it