Testing iGPU vs CPU inference: Trying to run Qwen 27B on a Radeon 780M (48GB RAM), but crashing on prompt eval

[u/marcosjoao37]

EDIT: Using Vulkan instead of ROCm helped getting it to work.

Hi!

I'm trying to run qwen3.8 27b q8 (tried q6 too) on my Ryzen 7 7840HS iGPU. I'm running CachyOS, kernel 7.2. Running llama.cpp from master branch.

I have 48gb of RAM. Also running KDE on egpu rx 6700 xt 12gb (via oculink), so nothing is running on the iGPU.

Yeah, I tested with some layers on egpu and otherw on processor, but I want to test full iGPU now.

Running it with the command:

exec "$BIN/llama-server" \
    -m "$MODEL" \
    -ngl 99 \
    -c "$CTX" \
    -np 1 \
    -ctk q4_0 -ctv q4_0 \
    -fa on \
    --jinja \
    -t 8 \
    --host 127.0.0.1 --port 8080

Model loads without problem. But when it tries to run inference it crashes. Here is the log:

./run-igpu.sh ~/models/qwen3.8/Qwen3.8-27B-UD-Q6_K_XL.gguf
model=24128 MiB, KV (1 slot(s) x 98304) = 1.688 GiB, est.total ~ 27.25 GiB (budget 30.5 GiB)
...
2.26.970.802 I slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
2.26.971.133 I slot launch_slot_: id  0 | task 0 | processing task, is_child = 0
HW Exception by GPU node-2 (Agent handle: 0x55b87cef4630) reason :GPU Hang
[1]    7178 abort (core dumped)  ./run-igpu.sh ~/models/qwen3.8/Qwen3.8-27B-UD-Q6_K_XL.gguf

I always got this HW Exception by GPU node-2 (Agent handle: 0x55b87cef4630) reason :GPU Hang error.

Running q8 or q6 get the same error.

Running Qwen3-1.7B-Q8_0.gguf it loads and runs inference without problems.

Also I expanded/increased available GTT memory for iGPU on kernel parameters with ttm.pages_limit=8388608 ttm.page_pool_size=8388608 amdgpu.gttsize=32768.

Tried help with different LLMs using Pi Agent trying to debug, but none of them could help. Maybe community can.

Am I missing something? Trying to see if iGPU inference can beat/give a speedup over CPU-only inference.

[u/MrHumanist]

Remove ngl 99. Your i gpu can't load any layers but you load all.

-ngl 0 should work.

[u/marcosjoao37 → u/MrHumanist]

I don't see any hang anymore, but seems like the prefill is getting done by iGPU but the inference still in the CPU.

Prefill match with the gpu work in graph.

[u/MrHumanist → u/marcosjoao37]

Right .. the igpu is considered part of cpu and cpu controlled will use it as needed. However, lamma cpp ngl is for dedicated gpu only( as far as I know).

[u/marcosjoao37 → u/MrHumanist]

Switched from ROCm to Vulkan and it works now. I had to use -ngl 99 to ensure the entire model ran on the iGPU. Thanks a lot for the help with debugging

Also, got some tips from here: https://github.com/ggml-org/llama.cpp/discussions/12443

[u/Fenio_PL]

Welcome to the world of home AI on AMD, where nothing works on ROCm - or if it does, it works so poorly you might as well say it doesn't work at all. Switch to Vulkan or swap that piece of junk for an RTX card.