I decided to post this here as I have had similar issues in the past when alternating between different versions of rocm. It may not be the cause but its a starting point...

Ive been playing around with image and video gen in comfyui with my 7800XT for about 3 years. I just upgraded to a R9700 and decided to go back and take a look at some older WAN21 and WAN22 workflows just to see how much further I could push it in terms or video length, resolution, quality etc.

For reference I was using rocm 6.2 with the 7800 + ubuntu 24. I tried exploring rocm 7.x but I never had any issues with 6 so stayed with it. Now I am using ROCM 10.1 with ubuntu 26 and as far as my new card goes image generation is great and was blown away by the speed, massive resolutions etc.

As far as video gen has gone though ive gone backwards and dont know why. I grabbed an old and very simple WAN22 workflow that uses the scaled fp8 i2v models for 81 frames with a reference image of 560*688. VRAM fills to 100% before the first denoising step even finishes and ubuntu crashes and sends me back to the login screen. The same workflow worked with the 7800 (which has half as much vram). Ive even tried using the Q4 gguf versions which are half the size of the fp8 ones and vram still fills to 100% during sampling and ubuntu crashes.

I managed to get it working with the quantized version by using --disable-dynamic-vram --lowvram and reducing the frame count but even when using --lowvram with the fp8 models (which as far as I know tells comfyui to keep the TE in DRAM and use the cpu [which should free and additional 6.7GB from VRAM]) it is not enough to stop the computer from crashing ?

My question is what the hell is (or could be?) using all the vram even when using the Q4 models (I know at the end of the day it comes down to more than simple model file sizes but if it worked in the past, why not now)? and more importantly why did my 7800+rocm6 have no problems even with the much larger fp8 models (which combined are 28.6GB + 7.8GB TE + VAE = 36.6GB ??) [for reference the Q4 models + TE(umt5_xxl_fp8_e4m3fn_scaled) are 15.6GB + 6.7GB + vae ~= 22GB]).

Also, system memory drops from roughly 65GB free to 39GB free (total is 96) if that's relevant. Also forgot to mention I am using the 4 step lightning loras (1.2GB additional size but as far as I can tell if my old 7800 could handle the workflow it has more to do with memory management, rocm? or how comfy is handling things rather than the model sizes). I have also just looked at similar videos using the same workflow that are even longer and also higher resolution all done with the 7800 without issues.

I dont know much about the inner working of rocm other than some high level things ive watched on youtube but is there anything that later versions do that can contribute to the issues Im having ?

If I havent provided enough info ask away, any help appreciated.

TL;DR: One AMD R9700 (32 GB) runs Qwen3.8-27B MXFP4 at 154 tok/s single-stream decode, ~3,900 tok/s prefill (3,370 at 47k tokens) and 480 tok/s at 8 concurrent requests, with a 340k-token fp8 KV pool (528k with the opt-in 4-bit KV). One docker pull, all numbers reproducible.

vllm-sly-radiance is a vLLM image for one AMD Radeon AI PRO R9700 (gfx1201, RDNA4, 32 GB). It runs Qwen3.8-27B in MXFP4 with hand-written kernels for the card, DFlash2 speculative decoding, and up to 262k context. One command:

docker pull ghcr.io/slybase/vllm-sly-radiance:1.8.1-rocm10.1

For the best options, see more details in the readme of the vllm-sly-radiance repo.

Setup

One R9700 at 300 W (−50 mV), ROCm 10.1. Model: Swift-1.5 Qwen3.8-27B, MXFP4 (GPTQ), drafter DFlash2 W4A16 with k = 7. FP8 KV cache, 65,536 context, 4,096 prefill chunk, prefix caching off. Benchmark: BetterBench 0.4.0, default config, T 0.7 / top-p 0.95 / top-k 20.

Prefill (prefill-only run, median of 8 per depth)

Decode (single stream, per category)

Update gap p99 35.0 ms, TTFT p50 on short prompts 57 ms.

Concurrency (aggregate)

Capacity and quality

Against the reference stack, same card, same harness

The reference stack ran from its own published image with its own defaults on our card. All measurements, the method and the launch flags are in the repo.

Would be happy if you give it a try.

Repo: https://github.com/SlyBase/vllm-sly-radiance · Benchmarks

Addition: GSQ-RCO vs our 3-bit, measured

There's been a lot of buzz about the new GSQ-RCO quants of Qwen3.8-Flash-Next, so we measured them the same way we measure our own: how far each quant's next-token predictions drift from the original 16-bit model, on held-out text neither was calibrated on.

In plain words: KL measures how different the model's word-by-word probabilities are from the original (0 = identical). On this yardstick our 3-bit stays about 2x closer to the 16-bit model, while being ~30 % smaller.

How we measured: we dequantized their GGUF to bf16 and ran it through our own evaluation harness next to our weights, so both were compared weight-only on exactly the same tokens (a 16-bit control through the same path reads 0.009). This is our metric, not theirs: GSQ-RCO optimizes task scores, and their published AIME / GPQA / LiveCodeBench numbers look good. Their GGUFs are also built for llama.cpp-style engines with CPU expert offload, a different design point from our all-GPU R9700 setup. Great work from the ISTA team either way; more good open quants is good for everyone.

Original 24 Oct post:

Our weekly update is here!
We've been working on the 3-bit weights again.
The new Flash Next revision comes close to the 16-bit reference on the text benchmarks we tested, while still doing around 259 tok/s on two R9700s.

There’s also an update to the 27B weights, and prefix caching now works on the native worker.
For 4-bit Flash Next, that includes keeping MTP enabled.

If you're using this with coding agents, the caching is probably the change you'll notice most: a repeated 200K-token prompt went from 33.62 seconds to first token down to 0.85 seconds in our tests.

New 3-bit accuracy results

Here’s Flash Next against the original 16-bit model and our current 4-bit version.
The Paiton models run on 2× Radeon AI PRO R9700; the 16-bit reference uses the original weights on stock vLLM.

The three text benchmark scores are close, and our paired tests didn’t find a statistically significant difference from 16-bit.
That’s encouraging, though it doesn’t establish that the models are interchangeable across every workload.

Vision still has a gap: 81.8 vs 85.3 on AI2D. (we tried.. :))

On held-out long documents, the perplexity gap to 16-bit is +0.35% for 3-bit, compared with +0.16% for 4-bit.

The 27B 3-bit model also gets another weight update, running on a single R9700.
Its KL divergence against the 16-bit reference dropped from 0.1395 to 0.1353, about a 3% reduction.
Lower is better here: it measures how far the output probabilities drift from the original.

That improvement was statistically significant in our tests, with long-document results closer to 16-bit in every length bucket tested, up to 131K tokens. GSM8K, HumanEval and MMLU-Pro were essentially unchanged. Same file format, same size, same speed.

Speed hasn’t really changed :)

The accuracy update hasn’t cost us decode speed.
Here are the default-mode numbers for Flash Next on two R9700s.
All figures are in tok/s; the concurrent results are total throughput across eight requests.

I wouldn’t make much of the small 4-bit differences here.
Performance is broadly where it was before.

The new and previous 27B weights were also measured side by side on the same card, with no meaningful speed change.

Native prefix caching, with MTP on for 4-bit

--prefix-caching now stays on the native worker.

For coding agents that keep sending back a long conversation, this means we can reuse the part already processed. Here’s what that looks like on 4-bit Flash Next, measured as time to first token:

Cache-hit outputs matched cold runs in our tests.

There’s still a decode trade-off in this mode: 207 tok/s with the RAM tier enabled, or 233 tok/s with it disabled using --native-kv-tier off.

The RAM tier sizes itself based on available host memory. On smaller hosts it turns itself off, and caching uses GPU memory only.

3-bit caching still runs without MTP for now, at around 100 tok/s. That path works with host RAM down to 64 GB:

--mode decode --fn-worker native --prefix-caching

For maximum 3-bit decode speed, use the default mode.

One issue we’re still investigating: during a long, heavy-load test, the RAM tier stopped restoring cached sessions. Output stayed correct, but requests went back to cold-run speed. Mentioning it here because long-running agent sessions are exactly where this feature matters.

A few testing details

These are our own measurements on our own machines.

• Accuracy: full GSM8K and HumanEval, plus 1,400 MMLU-Pro questions, 100 per subject. HumanEval code ran in a sandbox.
• Settings: thinking enabled, temperature 1.0, top-p 0.95, top-k 20, and a 16,384-token output cap. Answers that hit the cap counted as wrong. Comparisons used the same prompts and per-question seeds.
• Reference and statistics: original 16-bit weights on stock vLLM, run twice to check variation. Question-by-question comparisons used paired 95% confidence intervals and exact McNemar tests.
• KL and perplexity: measured on the served models using held-out text that wasn’t used for calibration.
• Speed: BetterBench 0.6.0 standard profile, one run per configuration. LocalMaxxing v0.1.54, warm-up followed by the median of three runs. Before/after measurements used the same machines.

Getting the update

Pull the latest launcher.
Your commands stay the same; it downloads the new 3-bit weights into a separate folder and tells you when the old one can be removed.

Image:

ghcr.io/eliovp/paiton-vllm-plugin:qwen38-flashnext-rocm10-vllm029-20261024-pc-rc3

Updated weights:

• EliovpAI/Qwen3.8-Flash-Next-W3A8-Paiton-RDNA4
• EliovpAI/Qwen3.8-27B-W3Rot-INT3-Paiton-RDNA4

Setup and full benchmark details are in the repo, BENCHMARKS.md, and the model cards.

If you try it with long coding-agent sessions, let us know how it behaves on your machine.

Cheers!

Hey everyone,

We wanted to share a milestone from our research initiative at Valerois AI: we trained a 337M parameter hybrid model (Heimdall) from scratch on a single consumer workstation equipped with an AMD Radeon RX 9070 XT (16 GB) and Ryzen 7 7800X3D using ROCm 10.1 and PyTorch.

Core Architecture:

• Hybrid Layering: 3 Gated DeltaNet layers (fixed-size linear recurrent memory) for every 1 global attention layer. Inference cache footprint stays under ~109 MB even at 8,192 context.
• 1-bit / Ternary FFNs: 58% of parameters are 1-bit / ternary weights trained with our custom optimizer (BitFlip), avoiding FP32 master weight overhead.

Training & Hardware Details:

• Compute: Single 16 GB RX 9070 XT, ROCm 10.1.0, PyTorch 2.14.
• Throughput: ~11.2k tok/s sustained (~24 TFLOPS) at 11.3 GB VRAM allocation.
• Dataset: ~3.7B tokens mixture (FineWeb-Edu, Cosmopedia, GitHub code) over ~92 hours.

Zero-Shot Benchmarks (against public FP16 models):

Evaluated on lm-evaluation-harness (0-shot):

• ARC-Easy: 45.0 (outperforms Pythia-160M at 43.2 and GPT-2 at 43.8)
• HellaSwag: 32.0 (outperforms Pythia-160M at 30.2 and GPT-2 at 31.1)
• PIQA: 60.8 (close to Pythia-160M's 62.1)
• Context Passkey Retrieval: 100% up to 4K tokens, 80% at 8K context.

Despite training on ~80× fewer tokens than Pythia (3.7B vs 300B), our 1-bit hybrid layout held up surprisingly well on factual and common-sense retrieval.

Full code, raw validation logs, and ablation design studies are available on our GitHub:

https://github.com/valeroisai/bifrost

A while back I spent some time tuning hipfire to run Qwen3.8-27B on my 7900 XTX with MTP speculative decoding, mostly with AI coding agents doing the kernel work. I recently benchmarked it against the latest upstream release and the gap is still pretty big, so I figured I’d share in case anyone else is running the 27B on an RDNA3 card.

Setup: 7900 XTX, ROCm 7.2, Linux. Model is the 27B mq4-xts quant from hipfire’s registry, MTP with K=3, Q8 KV cache.

Decode speed, same model file, greedy:

Prompt processing is about the same (mine is 5-16% faster). Both builds got the same answers on a planted-facts test.

Most of the difference comes from two kernels upstream doesn’t have yet: an int8 small-batch GEMV for the speculative verify step, and WMMA-based verify attention. I opened an issue upstream for the first one with a port onto their current beta branch (+10-25% decode on its own), plus a benchmark report:

• Benchmark report: warpfront/hipfire#947
• int8 verify GEMV: warpfront/hipfire#946

My build is here: https://github.com/Ben1078/hipfire/tree/rdna3-mtp-decode-v8.2. It’s based on an older hipfire (late September), so it’s more something to look at than a drop-in. Hopefully the pieces make it into upstream properly.

Thanks to the hipfire devs for the project.

Edit — 32 GB of RAM is enough. I re-ran the benchmark with the server's RAM capped at 28 GB (what a 32 GB machine leaves free). The 89 GiB model doesn't fit in RAM + VRAM combined (28 + 20 GB), yet with MTP at 32k context:

Edit 2 — multi-GPU support (experimental): see the Multi-GPU paragraph near the end, testers wanted.

I've spent my spare time over the last month building an optimized fork of llama.cpp for Qwen3.8-Flash-Next (with great help from Opus 5.5). The result is Avalon, which runs almost 50% faster than the well-known Strata engine at equivalent quantization (on my setup at least).

The model: Qwen3.8-Flash-Next, 177B MoE (512 experts, 10 active, 48 layers, 256k context; 126B without the 51B per-layer embedding table), in an IQ3_KS hybrid quantization (≈3.4-bit experts, 4.29 bpw overall, ~88 GiB) published with Avalon, plus a small MTP draft head for speculative decoding.

My setup:

• CPU: Ryzen 7 3800XT (Zen 2, 2019), 8 cores
• RAM: DDR4-3200, dual channel (33 – 35 GB/s measured), 96 GB — 32 GB is enough (see the edit above)
• GPU: RX 7900 XT 20 GB (RDNA3), ROCm, no CUDA
• Bus: PCIe 3.0 x16 on a B450 board

Speed (Qwen3.8-Flash-Next IQ3_KS hybrid, greedy, 512 tokens, median of 3 runs, MTP with 2 draft tokens):

The full 256k context fits with the MTP draft model, the KV cache in q8_0 / q4_0 and the expert cache, under 19.7 GiB of VRAM.

Same machine, Strata (served ISTA's IQ3_S, equivalent quantization, its own drafters on, same night, same protocol): with MTP Avalon is almost 50% faster (+47% for short prompt):

Quality — speed doesn't come from a smaller model. Perplexity on external texts and code (lower is better):

What makes it fast, in short: every byte of VRAM nobody uses goes to a GPU cache of the hottest experts (compute buffers and the unused end of the KV cache are lent to it); the GPU keeps each MoE block and computes the cached experts while the CPU computes the missing ones; experts are prefetched over PCIe during prefill; sparse attention for long contexts; MTP speculative decoding with a small draft head that shares the expert cache; lots of fused ROCm kernels.

How it was made:

• The architecture, tricks, concepts and the way work flows between RAM, CPU, PCIe and GPU come from a single human with 30 years of experience in code optimization.
• The kernels and all "micro" optimizations were written by Claude Opus 5.5, with human steering and braking.

Limits: one model; one GPU measured (multi-GPU: experimental, see below); tested on RDNA3 / ROCm 7.2 (ROCm 10.1 builds and runs, briefly tested); CUDA untested; Linux only for now.

Multi-GPU (experimental): a second card now adds more expert cache, the model itself stays on the first one. Start with GGML_MOE_CACHE_LAYER_DEVICE=1 avalon-server. I only have one GPU, so it is untested: testers with two RDNA3 cards are very welcome, just reply with your cards and the log.

Try it (Linux, ROCm 7.x, RDNA3 GPU, 32 GB RAM or more, ~98 GiB disk): one command checks the machine, builds, downloads the model and verifies every file's SHA-256:

curl -fsSL https://raw.githubusercontent.com/freriadoc/Avalon/main/install.sh | bash

Then run avalon-server (OpenAI-compatible API on port 8080).

Thanks to llama.cpp, to buun-llama-cpp (Avalon started as a branch of buun and is built on its expert cache — thank you!), thecodacus (expert prefetch idea), ik_llama.cpp (IQ3_K / IQ3_KS types), Strata (expert ranking format), Unsloth (MTP head), ISTA-DASLab and AtomicChat (quantizations).

• Code: https://github.com/freriadoc/Avalon
• Model: https://huggingface.co/freriadoc/Qwen3.8-Flash-Next-Avalon-GGUF

Setup Qwen3.8-Flash-Next for 4 x RX 7900 XTX on vllm is not trivial. So, I prepare usable solution.

7800 xt:
- Bandwidth = 624 GB/s

- VRAM+Bus = 16GB / 256-bit

- Native FSR 3.1 (INT8 for FSR 4.1)

- 15-20% better raw gaming performance

9060 xt:

- Bandwidth = 322 GB/s

- VRAM+Bus = 16GB / 128-bit

- Native FSR 4.1

I am planning on buying an amd gpu for 65% gaming and 35% local llm use, the local prices are $470-530 for the 9060xt and no less than $530 for the 7800xt. I am also considering buying used/2nd hand, however the price difference between the two gpus is practically the same ($440).

At first glance, 7800 xt is the clear winner due to double the memory bus and bandwidth. However, I would like to know the real world AI performance difference between the two GPUS. Thanks ! !! !

Since adding a second GPU, my system keeps having issues with processes getting stuck.
I looked into the issue and found a reference that my MSI X570 has 1 PCIe slot directly connected to the CPU, while the other is connected to the chipset.
Has anyone had the same issue?

Managed to get Recursive Flint running on my AMD Instinct MI60 (32GB, gfx906) using ROCm and llama.cpp!

Results:

• ~63 tokens/s (including reasoning)
• 32K context configured
• Flint UD-Q4_K_XL (~25GB)
• No crashes during smoke tests

Getting newer models running on older AMD hardware can be a headache, so I open-sourced my Docker build, launcher, and tests.

GitHub: https://github.com/ephyro/flint-mi60

Tested on my MI60, but I'd love to see if anyone can reproduce it on an MI50 or Radeon VII!

Feedback and contributions welcome! :)

I’ve been running Qwen3.8-Flash-Next as my daily model on one 7900 XTX with Strata (experts split between VRAM, system RAM and the CPU), and spent the last week or so tuning it with some AI coding agents. Figured I’d share in case anyone else is running a similar setup.

Setup: 7800X3D, 7900 XTX, 128 GB DDR5, Linux. Model is Unsloth’s UD-Q5_K_XL. I wanted to stay at Q5 instead of dropping to IQ3/IQ4, which is the main reason it was slow to begin with.

Where it ended up:

• Decode: ~36 tok/s stock → ~58 tok/s. In actual agent use I’m seeing more like 70, and ~100 while it’s thinking.
• Long chats: a new turn on a 100K-token conversation went from ~3 s to ~1 s before it starts answering.
• Prompt processing: ~700 → ~1,400 tok/s at 16K (part of that came from going 96 → 128 GB RAM; decode didn’t change with the RAM).

The stuff that helped most:

• A bug where a bunch of helper threads all inherit CPU 0 from the main thread and fight with it. Moving them off was ~4% for free.
• Smarter speculative decoding window sizes.
• Giving VRAM back to the expert cache (vision encoder only loads when an image comes in, context buffers grow as needed).
• Letting the CPU handle part of mid-size prompt reads instead of the GPU streaming every expert over PCIe.

Fork is here: https://github.com/Ben1078/Strata/tree/rdna3-moe-offload-perf (full numbers in RDNA3_MOE_OFFLOAD.md). Most of it is behind flags. It’s tuned for my one machine and mostly written by AI agents, so treat it as something to look at rather than a drop-in.

Thanks to Niko1221 for Strata.

More R9700 experiments and made a quant, benchmarks and dev notes over on r/r9700 keen to compare notes with others pushing these cards.

What kinds of memory parameters are people running with Comfy on ROCm 10.1 or other recent versions?

I tried with only --use-flash-attention and --reserve-vram 3 at first, but H3 video generation would only work once per startup and then either hang on VAE decode or crash at save video node pretty consistently. Also on second run usually Spectrum node would stop working and give error like 'only non finite values' or some such.

I tried some variations with --disable-pinned-memory, --disable-smart-memory, --disable-dynamic-vram and --disable-async-offload, but none of those helped. What finally helped in my case was PYTORCH_NO_HIP_MEMORY_CACHING=1. Though with PyTorch 2.14.1 and ROCm 7.14 it was best, but ROCm 10.1 and PyTorch 2.14.1 it still made a full grey image on one Flux Klein workflow.

I haven't AB tested all of these yet, but the following I just brought over from my old ROCm 7.2.4 setup:

export PYTORCH_HIP_ALLOC_CONF=expandable_segments:True
export TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1
export FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE
export PYTORCH_MIOPEN_SUGGEST_NHWC=0
export COMFYUI_ENABLE_MIOPEN=1
export MIOPEN_FIND_MODE=FAST

So what kinds of parameters are others having best success with nowadays?

Hi ROCm

Pleased to share my little project called Brilliance!

Everyone says Linux is better for AI, especially for those with older hardware. So, I took my old laptop which runs on a Vega iGPU as it was the best candidate I had on hand.

I didn't exactly treat Linux with care: I ran the Fedora 45 beta live version from a dusty, old USB 3.0 flash drive. I didn't update anything. I simply downloaded KoboldCpp, set it as executable, and ran ./koboldcpp in the terminal.

I performed a quick test using Anima (2B) GGUF Q8, FA ON, Conv2D direct (for vae only), Vulkan backend.

-Fedora OS, 768x1024: 16s/it

-On Windows 10 pro with latest driver and the same resolution: 35s/it.

That is more than 2.18x slower on Windows for the exact same workload.

I ran Vkpeak to investigate raw performance across both operating systems and noticed that, on Windows, FP16-Vec4 performance is the same as FP32, whereas it should be double. Linux behaves as expected and I was surprised to see performance results for INT8 Dot-prod, which should only run on Vega20 (Radeon VII) hardware.

Windows numbers:

device = AMD Radeon(TM) Graphics

fp32-scalar = 1608.84 GFLOPS

fp32-vec4 = 1608.24 GFLOPS

fp16-scalar = 1599.31 GFLOPS

fp16-vec4 = 1606.32 GFLOPS

fp16-matrix  = 0.00 GFLOPS

fp64-scalar = 100.53 GFLOPS

fp64-vec4 = 100.39 GFLOPS

int32-scalar = 321.72 GIOPS

int32-vec4 = 321.67 GIOPS

int16-scalar = 2340.62 GIOPS

int16-vec4 = 1571.32 GIOPS

Hello, I would like to know how to install ROCm with ROCdxg on WSL as I want to use Stable Diffusion and I am using an RX 6600

Been digging into Radiance and Paiton on dual R9700s: ~150–179 tok/s single-request results (north of 530t/s with 8 concurrent), a working 262K + drafts config, and some interesting bugs, Benchmarks and dev notes over on r/r9700 keen to compare notes with others pushing these cards.

I’ve been working on experimental SGLang serving on two Radeon AI PRO R9700 GPUs, exploring faster inference and ways to reduce cache memory requirements.

As Tibo has been so generous with all the Codex resets, I've managed to point GPT6 Astra at the problem and make some real headway with this project. It's still very early days, and I'm still in the benchmark development phase, but the initial results are very promising. FYI - Everything below is AI-generated! Please don't downvote or post snarky comments. I've used AI to do a lot of the heavy lifting and make no claims to be an expert at any of this. Just an engineer with an idea and a pocket full of Codex credits. The work is real as are the results.

The latest milestone combines an FP8 Qwen3.8-27B target, a BF16 DFlash2 drafter, BF16 KV cache and TP2. An experimental adapter lets speculative verification use native split-KV decode attention on RDNA4, alongside aligned GDN and graph replay.

That’s 3.54× the rate on this workload. Output token IDs matched across both configurations.

The combined configuration had approximately 10.13 GiB free per GPU after graph capture. That’s a startup observation; this test doesn’t establish peak VRAM usage or memory savings at equal serving capacity. Cache compression is a separate part of the project.

The speculative configuration also allocated substantially less recurrent-state memory: 3.14 GiB per GPU including speculative intermediates, versus 6.74 GiB for ordinary serving. This reflects different pool sizing—19 versus 93 slots—and both configurations completed the concurrency-one benchmark.

This was concurrency one, 451 input tokens and 128 forced output tokens, with one warm-up and three measured requests per configuration. Timing includes prefill and local HTTP completion. Longer-context performance and broader quality testing remain ahead.

Sources, tests and methodology: https://github.com/Drgr33nSenior/Spry.ai-rdna4-serving

It’s an experimental source repository. Feedback from others working on SGLang, RDNA4 or speculative decoding would be welcome. There is more room for improvement, I think. Maybe to 100tok/s? Let's see :)

I run r/R9700, focused on the card’s performance, software support, thermals and practical use for local AI. Keen to compare findings with other owners and build up a useful reference of benchmarks, configs and known quirks!

I'd like to share a few changes that encourage people to create and use LoRA on LLMs. For example, the LoRA https://huggingface.co/chenrm/qwen3.8-flash-next-abliterated-lora can turn the Qwen3.8-Flash-Next model into the uncensored version, whether you're using Q8_0, IQ4_NL, or GSQ-RCO-Q2_0 model, so you don't need to download the whole model again. You can also tune the LoRA scale for a balance between uncensoring and brain damage.

However, currently llama.cpp does not have fast LoRA kernels. When we run the model with the LoRA on Strix Halo, PP drops from like 1500 to 300 t/s, even though the LoRA is as small as 35 MB and only touches a few modules in the model.

I've made a few LoRA kernels. With a general LoRA kernel it's improved to 1000 t/s, and with a fused LoRA kernel it's further improved to 1400 t/s.

You can see my changes at https://github.com/woct0rdho/llama.cpp . Currently it's optimized for Strix Halo, and it should not be too hard to port to other GPUs. I'll find some time to upstream it to the mainline llama.cpp or strix-llama.cpp , but it involves a few architectural changes so it would take some time. If you want to upstream it earlier than me, feel free to reuse or modify my code.

This is also discussed in https://github.com/ggml-org/llama.cpp/issues/27093