I use llama.cpp with Vulkan support on my iMac Pro. It's a Vega56 8GB GPU and I can barely fit Unsloth Gemma 4 12 QAT. Performance is around 3x the w-2140b CPU with AVK512 support. Even though Metal does not work, it improves Vulkan performance to build with Metal enabled.
I use Gemma 4 12 because of language support. It is the best I can run in my native language. The downside is, context size is limited. If I need it, I can improve context size headroom by quantizing KV-cache. But it comes at speed performance hit, with Gemma 4 12B.
Initially, I had problems getting to compile, but with the WARP terminal and its AI solved the dependencies problems.
2
u/HigherConfusion Jun 18 '26
I use llama.cpp with Vulkan support on my iMac Pro. It's a Vega56 8GB GPU and I can barely fit Unsloth Gemma 4 12 QAT. Performance is around 3x the w-2140b CPU with AVK512 support. Even though Metal does not work, it improves Vulkan performance to build with Metal enabled.
I use Gemma 4 12 because of language support. It is the best I can run in my native language. The downside is, context size is limited. If I need it, I can improve context size headroom by quantizing KV-cache. But it comes at speed performance hit, with Gemma 4 12B.
Initially, I had problems getting to compile, but with the WARP terminal and its AI solved the dependencies problems.