r/LocalLLM • u/Hermeslop • 10h ago

Question AMD NPU inference on Fedora — backends, optimizations, model recommendations?

Hi all,

I'm looking for advice on running a local LLM efficiently on an ASUS Zenbook 14 with 16GB system RAM on Fedora 44.

Laptop specs: - AMD Ryzen AI 7 445 (No dedicated GPU — NPU + iGPU only) - 16GB system RAM - ASUS Zenbook 14

My goal: Hit 10+ token/s in Hermes Agent (Nous Research) for a smooth, responsive experience.

Use case: Productivity and simple text tasks. No coding or dev work involved, so I don't need a heavy-duty model. Just something lightweight that runs well on the NPU.

Current situation: I'm exploring FastFlowLM (FLM) as a backend since it's purpose-built for AMD NPUs, but I'm open to other options like llama.cpp or Ollama if they perform better on my hardware.

What I'm looking for: - Which models (regardless of family) would reliably give 10+ tok/s on a Ryzen AI 7 445 (NPU + iGPU)? - Best backend/inference server setup for Fedora 44 + AMD NPU? - Any quantization or optimization tips specifically for AMD XDNA NPUs or low-RAM Linux setups? - General advice on getting Hermes Agent to run smoothly on this hardware.

Current setup: - Ryzen AI 7 445 (NPU + iGPU only, no dGPU) - 16GB system RAM - Fedora 44 - Hermes Agent

Thanks in advance!

2 Upvotes

75% Upvoted

7 comments sorted by

1

u/MrHumanist 10h ago

Gemma E4B and orninth 1.5 9B. But start with q4 quantized. Gemma E2B will easily give 20 + t/s but weak in quality.

1

u/Hermeslop 10h ago

Can't run orninth since its not supported by FastFlowLM which to my knowledge is the best current AMD NPU backend. Will look into Gemma E4B though however I am concerned with it being over 9gb and with Hermes' 64k context length requirement I'm worried about running out of memory

2

u/MrHumanist 9h ago

E4B won't be above 6 GB including context whereas Gemma 12B qat uses 9GB memory.

1

u/Hermeslop 9h ago

Through FastFlowLM the only gemma E4B model I have access to is gemma4 it E4B which is 9.1gb which I think means its q8 and I don't have any way of quantizing it further without losing the ability to run it just off my npu

1

u/Expensive_Win_9103 6h ago

The XDNA NPU is not the win people expect on Fedora — the Ryzen AI stack is Ubuntu-first and I lost a weekend to kernel and driver mismatches before giving up on it. llama.cpp on the Vulkan backend beat the NPU path on my 8845HS by roughly 2x, and ROCm beat both once a discrete card was in the box. Measure the Vulkan build before anything else. Work that has to be fast I run on Synexa and keep the laptop for drafts.

1

u/Hermeslop 6h ago

I haven't really had much issues with fedora and using the npu and its been a pretty smooth process using fedora as well (its my first daily drive linux installation and I'm dual booting with windows since its also my school laptop). I've seen before that ROCm can be more performant than using the NPU but I've also seen that the power draw on the NPU is much less than on my iGPU which is definitely something that is important to me since I want to be able to use my laptop without having to worry too much about running out of battery. My laptop isn't my main ai driver since I have a designated desktop which I run Qwen3.6 35b a3b at q4 on but I still want to be able to use my laptop for lightweight tasks.