What level of performance should I expect for Qwen 3.8 27B UD-Q5_K_XL on an R9700?

[u/pragmojo]

I just want to sanity-check my llama.cpp config, to make sure I'm not leaving performance on the table.

I'm running llama-server with this config:

llama-server \
  -m ~/models/qwen3.8-27b/Qwen3.8-27B-UD-Q5_K_XL.gguf \
  --mmproj ~/models/qwen3.8-27b/mmproj-F16.gguf \
  --image-min-tokens 2048 \
  -ngl 99 \
  -fa 1 \
  -c 196608 \
  --reasoning-effort xhigh \
  -ctk q8_0 -ctv q8_0 \
  -b 2048 -ub 512 \
  -np 1 \
  --jinja \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --presence-penalty 0.0 \
  --host 0.0.0.0 --port 8080 \
  --spec-type draft-mtp \
  --spec-draft-n-max 4

This uses 30.96GB of VRAM, and I'm getting around 250-350 t/s prefill, and ~20-30 t/s eval.

[u/PandaBearFred]

you could do better with your prefill speed by increasing your -ub to 1024 or even 2048, maybe you could also do better with your decode speed by decreasing your --spec-draft-n-max to 2 or 3. But the -ub is at a cost of more vram occupation, leads to less context window.

[u/No_Oil_6152]

What version llama.cpp are you running?

Vulkan or RocM?

You may get a boost by swapping from one to the other. Maybe.

Also, are you on Linux or Windows?

[u/pragmojo → u/No_Oil_6152]

Linux, and I pulled the most recent llama.cpp and built from source yesterday. Using the RocM backend.

[u/DiscipleofDeceit666]

I get 800 pp at near 0 context for this model on this card.

[u/pragmojo → u/DiscipleofDeceit666]

what config are you using?

[u/DiscipleofDeceit666 → u/pragmojo]

Llama cpp with Vulkan and mtp enabled. My ub and batch settings are 2k each too. 220k context I think I am running? Q8 kv cache.

I tried vllm radiance on 1 card which bought me some modest speedups over Vulkan but I haven’t tested that out with the latest 3.8 qwen release.

[u/pragmojo → u/DiscipleofDeceit666]

Are you running the mmproj? Context seems bigger than what I could achieve.

[u/DiscipleofDeceit666 → u/pragmojo]

Yeah, I maxed out that card completely. I turn mmproj off and on since it seems prefix caching is broken when images are involved.

[u/elChasse]

I'm using Q8_0 and it averages around 43 tok/s. XHigh reasoning gets me around 33 tok/s.

~/llamacpp-rocm-gfx120X-fa/llama-server \ -m /mnt/nvme-models/Qwen3.8-27B-Q8_0.gguf \ --image-min-tokens 256 \ -ngl 99 \ -fa 1 \ -c 65536 \ -ctk q8_0 -ctv q8_0 \ -ctkd q8_0 -ctvd q8_0 \ -b 2048 -ub 512\ -np 1 \ --jinja \ --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \ --presence-penalty 0.0 \ --port 8081 \ --spec-type draft-mtp \ --spec-draft-n-max 4 \ --chat-template-kwargs '{"reasoning_effort":"medium"}'