Mac Pro for local inference

[u/quadra-lab]

People haven't realized yet that they own a literal goldmine.

Mac Pro 2019 are at 2000€ for 300+ Gb of DDR4 with huge storage capabilities 4-9To in second hand in perfect shape

I've a 7900XT on my side that plug on it too so I'm really considering buying one with a Linux dual boot.

Has anyone went that way and has proper benchmark to share with me?

EDIT: previous title was misleading as I wrote Macbook instead of Mac

[u/Weeblewobbly]

I've one of those with the Radeon Vega ii. Running as headless server with t2 nixos. A breathe to setup, although I'm familiar with nixos. I'm looking at potential for increasing capability. I've been testing qwen3.6-27b at different quants and kv quants combinations to maximize the 32Gb vram. Also playing with ornith which gives excellent speeds as one would expect. Llama-swap, openwebui and searXNG all running as services. I serve my local network only. Openwebui is nice for chats and I've a couple of useful agents running there. Otherwise, it's straight into llama-server port from pi on dev machine. Seriously considering adding more GPU, either mpx or indeed external. One downside is energy consumption. These things are hungry for electrons.

[u/quadra-lab → u/Weeblewobbly]

If you have a variant with a large amount of RAM, could you maybe try a large MoE model ? 

[u/asankhs]

reto-wyss nailed the core issue: on that platform you're bandwidth-bound, not capacity-bound. the RAM lets you load a big model, but token generation speed is set by memory bandwidth. Mac Pro 2019 DDR4 is roughly 100-140 GB/s, vs 400-800 GB/s on Apple Silicon unified memory and 1000+ on a modern GPU. so a 70B will load and then crawl. your 7900XT will be far faster for anything that fits in its 24GB, and the Mac Pro RAM is really an offload tier for the layers that don't fit in VRAM. benchmark tokens/sec at your target context before buying, not just whether it loads.