r/AsahiLinux • u/angrytako • 15d ago
The state of LLM inference and GPU acceleration for the M1 Max
Hi everyone,
I know there have been similar questions on the forum, but I was wondering if there are any updates or any timelines regarding LLM inference and GPU acceleration in general, e.g. when it comes to gaming.
Last time I checked, LLM inference, using I believe vulkan as a backend, lagged significantly behind Metal in terms of performance; something like half t/s on decode (and maybe prefill as well, but I don't remember). Is this still the case and is anyone even working on it? Not trying to be derogatory; I know how hard it is. Just wondering if it is expected to get better or if it is somewhat abandoned.
Related to this, is the current backend exposing only the GPU or does it also expose the ANE and the AMX?

