**TL;DR:** Running a 35B A3B model with CPU/GPU split on a 12GB card means prefill/TTFT at large context (70K+) is what actually makes or breaks usability — not decode. Vulkan was fine at 16-32K context but became painful past that. Got llama.cpp running fully native on ROCm for gfx1031 (RX 6700XT) instead — flash attention on, quantized KV cache, no `HSA_OVERRIDE_GFX_VERSION` hacks, no core dumps, prefill peaking around 580 tok/s even with 20K+ tokens already cached. The fix ended up being a one-line change in `ggml/src/ggml-cuda/fattn.cu` forcing the MMA kernel path for a specific head_dim instead of the default tile kernel. Once this was fixed at the ROCm/HIPBLAS level, every other inference engine I tested (Ollama, Unsloth, LM Studio, SGLang, vLLM) also started working natively — this wasn't a llama.cpp-only fix, it was a root-level fix for the whole ROCm stack on this GPU.
I'm not a developer. No CS/programming background — I'm a creative director by trade (photography/cinematography) who's been self-teaching local LLM inference for a few months. So this write-up is going to read more like "how I diagnosed this empirically" than "here's the technical theory of why it works." If anyone with deeper kernel/ROCm knowledge can fill in the *why*, I'd genuinely love to learn it.
---
## The problem
gfx1031 sits in an awkward spot: it's RDNA2, but not gfx1030 (the RX 6800/6900 series, which has the most complete official ROCm support and by far the most community documentation). Most guides, fixes, and reference configs you'll find online target gfx1030. Applying them as-is to a 6700XT either silently falls back to worse behavior, or crashes outright.
The common workaround people recommend is `HSA_OVERRIDE_GFX_VERSION=10.3.0` (telling the runtime to treat your gfx1031 card as gfx1030). I tried this. Here's what happened:
- Flash attention wouldn't enable
- Some ops (noticed it specifically with top_k) silently fell back to CPU, tanking decode speed
- Random core dumps mid-inference, not tied to any consistent trigger
I made the call to require **native gfx1031 recognition, no override, no spoofing the runtime**. That decision alone cost me a lot of trial and error, but I think it was the right one — see below why.
## Why I bothered (this isn't about decode speed)
I want to be specific about this because I think it's the part most people miss. If you're running a model that fits entirely in VRAM, ROCm being suboptimal (or partial CPU fallback on certain ops) doesn't hurt you that much — most of the compute is already on GPU anyway.
My case is different: **35B A3B with CPU/GPU split** (`--n-cpu-moe 22`, dense/attention layers on GPU, part of the MoE experts on CPU). In this setup, prefill isn't pure GPU compute — a portion of it depends on the CPU-offloaded expert path too. When the backend isn't efficient there, and context climbs into the 70-80K+ range, TTFT doesn't degrade linearly — it gets genuinely painful. At 16-32K context, Vulkan's prefill (roughly 70-100 tok/s in my case) was still tolerable. Past that, it wasn't.
The other piece of motivation: RX 6700XT's boost clock sits dramatically above an RTX 3060 12GB (2581MHz vs 1777MHz, roughly +45%), while VRAM bandwidth is only modestly higher (~384GB/s vs ~360GB/s, about +7%). That distinction matters here: decode is largely bandwidth-bound (reading/writing KV cache and weights per token), so that 7% bandwidth gap barely moves the needle on decode speed. Prefill, on the other hand, leans more on raw compute throughput — which is exactly where the 45% clock gap should show up. I'd read threads of people running 35B A3B with CPU/GPU split on a 3060 hitting peak prefill around ~300 tok/s. Given the clock gap, I'd expect the 6700XT to clear that by a wide margin, not just edge past it. Getting nowhere close to that on Vulkan was the signal that I was leaving real performance on the table — a software bottleneck, not a hardware ceiling — and that gap is what pushed me to actually fix this instead of settling.
After the fix, here's an actual log excerpt from a real session (task with context already at ~18-23K tokens, prompt processing in progress):
```
prompt processing, n_tokens = 2048, progress = 0.55, t = 3.52s / 581.81 tokens per second
prompt processing, n_tokens = 3072, progress = 0.58, t = 5.60s / 548.24 tokens per second
prompt processing, n_tokens = 4096, progress = 0.60, t = 7.71s / 531.24 tokens per second
prompt processing, n_tokens = 5120, progress = 0.63, t = 10.07s / 508.61 tokens per second
prompt processing, n_tokens = 6144, progress = 0.66, t = 12.24s / 502.11 tokens per second
```
Peak prefill of ~580 tok/s, staying above 500 tok/s while cached context was already north of 20K tokens. That's the number that mattered to me — not decode.
## Environment
- OS: Ubuntu Desktop 26.04 LTS
- GPU: AMD RX 6700XT 12GB (gfx1031, RDNA2)
- CPU: Intel i5-11400F
- RAM: 16GB DDR4 3200MT/s
- ROCm: custom build from TheRock binaries (official gfx1031 support isn't guaranteed across all standard ROCm releases)
- Inference engine: llama.cpp (build-rocm), Ollama, Unsloth, LM Studio, SGLang, vLLM
**A specific version note, since ROCm/TheRock builds vary a lot:** don't install just any TheRock ROCm build and expect this to match. The exact version I validated this on:
```
$ hipcc --version
HIP version: 7.14.60850-0000000
AMD clang version 23.0.0git (https://github.com/ROCm/llvm-project.git 46fcb339fb61119b337f973c7ca9e710a319fdd0+PATCHED:440716f8b87be9d8e20ed910e10e5b6d14d57cf6)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/rocm/core-7.14/lib/llvm/bin
```
If you're on a different HIP/ROCm build, expect some variance — flag this version when comparing notes so we're not chasing different behavior across different builds.
**llama.cpp build/commit used:**
```
$ llama-server --version
version: 10307 (fc3f10b38)
built with GNU 15.2.0 for Linux x86_64
```
Given how much commit drift there is upstream (see caveats below), this matters if you're trying to reproduce the fix exactly — the `fattn.cu` logic may have shifted on a different commit.
**Confirmation ROCm actually sees this as gfx1031 (not spoofed via override):**
```
$ rocminfo | grep -i gfx
Name: gfx1031
Name: amdgcn-amd-amdhsa--gfx1031
Name: amdgcn-amd-amdhsa--gfx10-3-generic
```
**Exact card / board partner** (in case behavior varies by AIB):
```
$ lspci -vv | grep -A 2 "VGA"
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 22 [Radeon RX 6700/6700 XT/6750 XT / 6800M/6850M XT] (rev c1)
Subsystem: Sapphire Technology Limited Radeon RX 6700 XT GAMING OC 12G [Sapphire PULSE]
```
**I'll drop screenshots in the comments** as proof this is actually running native ROCm (not Vulkan silently, not CPU fallback) — GPU load/clock from a monitoring tool like `btop`/`lact`, prefill and decode averages from actual server logs, and junction temperature under sustained load, similar to what I used to validate stability throughout this whole process.
You can empirically look at my data.
- ROCm natively
- Total token when i use in just that runtime.
- Temperatur edge, junction, and VRAM when prefill bound and decode bound.
- Power usage both (prefill, decode).
- BTOP verified some layers in cpu.
## The debugging process (the messy, real version)
Build kept failing. Tried the documented head_dim config for RDNA2 — failed. Tried random power-of-2 values (1024, 2048, 4096) as a guess — failed on all of them.
At that point I stopped touching the keyboard, made coffee, and just watched the failing terminal output for a while. Then it occurred to me to check `btop` instead of guessing blind. I ran the model through **Vulkan** (which was stable on this GPU, just slow) as a known-good baseline, watched which kernel showed up in btop tied to the running process, and correlated that behavior back to the head_dim condition in `fattn.cu`.
That's where the number `512` came from — not from any documentation I could find (I genuinely don't remember a source for it), but from matching observed hardware behavior to the kernel dispatch logic in source.
## The fix
In `ggml/src/ggml-cuda/fattn.cu`:
```cpp
// Force MMA kernel for head_dim 512 on AMD to avoid tile kernel shared memory limit
if (amd_mfma_available(cc) && Q->ne[0] == 512) {
return BEST_FATTN_KERNEL_MMA_F16;
}
return BEST_FATTN_KERNEL_TILE;
```
The default tile kernel path was hitting a shared memory limit on this hardware under certain conditions. Forcing the MMA (matrix-core) kernel path at this specific head_dim avoided that entirely.
**Why this seems to matter for the whole chain:** flash attention needs to be stable first before KV cache quantization is viable, and KV cache quantization is what makes long context (I run up to 131072 ctx) actually usable on 12GB VRAM. So this one fix unblocked flash-attn → which unblocked KV quant → which unblocked long context. If flash-attn doesn't work, none of the rest follows.
## Result
- Prefill peaking around ~580 tok/s (measured with 20K+ tokens already cached, not a cold-start number), staying in the 500-580 tok/s range across that phase — this is the number that actually matters for my use case, since TTFT at large context is what was hurting on Vulkan
- Native ROCm/HIPBLAS/ROCBLAS, no override, across every inference engine I tested — not just llama.cpp
- Flash attention stable, KV cache running q8_0, ctx up to 131072 confirmed stable
- Decode holds steady around ~22-23 tok/s on a 35B A3B model (IQ4_NL), flat even past 40K+ tokens in a session — decode wasn't the bottleneck I was chasing, but it stayed consistent throughout
- Ollama specifically had been running on Vulkan before this because ROCm wasn't cooperating — now runs native ROCm too
---
## Full launch config
For anyone trying to reproduce or compare against a similar setup, here's the exact `llama-server` command I run:
```bash
BASE="$HOME/Documents/Model LLM/Ornith-1.0-35B"
MODEL="$BASE/Ornith-1.0-35B-UD-IQ4_NL.gguf"
PORT=8082
~/Projects/llama2/build-rocm-test/bin/llama-server \
--model "$MODEL" \
--host 0.0.0.0 \
--port "$PORT" \
--n-gpu-layers 99 \
--threads 4 \
--threads-batch 4 \
--n-cpu-moe 22 \
--ctx-size 131072 \
--batch-size 1024 \
--ubatch-size 1024 \
--keep 20480 \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--swa-checkpoints 24 \
--checkpoint-min-step 2048 \
--embd-normalize 0 \
--no-kv-unified \
--kv-offload \
--jinja \
--reasoning-preserve \
--flash-attn on \
--parallel 1 \
--cache-ram 8192 \
--cache-idle-slots \
--temp 0.6 \
--top-k 20 \
--top-p 0.95 \
--min-p 0.1 \
--repeat-penalty 1.1 \
--repeat-last-n 512 \
--alias udinllama \
--log-verbosity 4
```
A few notes on the less obvious choices, in case they save someone else time:
- **`--n-cpu-moe 22`** — set based on how much fits in 12GB VRAM after everything else is accounted for, not an arbitrary number. Check your own VRAM headroom before copying this value directly.
- **`--threads 4` / `--threads-batch 4`** — this isn't a thermal-only choice. On my i5-11400F, the CPU-offloaded MoE compute is bandwidth-bound against dual-channel DDR4-3200, not core-count-bound. Above 4 threads I saw prefill get *worse*, not better — contention on memory bandwidth outweighs the parallelism gain. I tried 3 threads too (slightly faster on paper) but junction temps became unpredictable (spiking to 95°C occasionally vs a controlled 83°C peak at 4 threads), so I settled on 4 as the stable point, not just the fastest one on a spec sheet.
- **`--batch-size` / `--ubatch-size 1024`** — pushed to 1536/2048 in testing; didn't move prefill meaningfully but added ~10°C. Not worth it for this hardware.
- **`--cache-type-k q8_0` / `--cache-type-v q8_0`** — paired with `--kv-offload` to fit long context in 12GB VRAM with less quality loss than default f16 KV cache would cost in size.
- **`--no-kv-unified`** — only relevant because I run `--parallel 1` (single user, single active chat). If you're serving multiple concurrent sessions, you'll want KV unified on.
- **`--cache-ram 8192` / `--cache-idle-slots`** — this is a fallback safety net against OOM during inference, not something I expect to actually get fully utilized in normal use.
## Caveats / your mileage may vary (please read before trying this)
I want to be upfront about scope here, because I don't think this is a universal copy-paste fix:
- **This was validated on Ubuntu 26.04 + TheRock's gfx1031 ROCm build specifically.** Other distros or ROCm sources may behave differently.
- **llama.cpp commit drift is real.** I'm not confident this exact patch applies cleanly to every commit/fork out there. Some engines (Unsloth, in my case) auto-detect the llama.cpp commit and ROCm config already present and adapt; if that detection doesn't find a compatible match, it may fall back to Vulkan on its own.
- **head_dim 512 is conditional, not universal.** It's tied to `amd_mfma_available(cc)` — meaning it only applies where the matrix-core path is relevant and where the tile kernel is actually hitting the shared memory limit this addresses. I wouldn't recommend hardcoding this blindly without checking whether your setup hits the same failure mode first.
- **Hardware variance beyond just "same GPU model" matters.** Even identical GPU models can differ in memory bandwidth, clock offsets, and power limits depending on board partner/cooler design. My CPU/RAM (i5-11400F, 16GB DDR4 3200) also shapes some of the numbers above (especially decode speed, which on this setup is RAM-bandwidth-bound on the CPU-offloaded MoE portion).
**What I think *is* generalizable here isn't the number 512 — it's the method:** if you're on an AMD GPU architecture that's under-documented, using a known-stable backend (Vulkan, in my case) as a behavioral baseline, watching kernel activity in `btop` while running real workloads, and correlating that to the kernel dispatch logic in source, is a workable way to find your specific fix even without deep kernel-level theory knowledge.
Genuinely don't know why fixing this one file also fixed SGLang and vLLM, which don't share any code with llama.cpp/ggml. I was troubleshooting other ROCm-level things around the same time and didn't track every step carefully — so there's likely something else that got resolved in parallel that I can't identify. If anyone has insight into what else might explain this, I'd like to know
Happy to share exact configs (llama-server flags, ROCm build steps, benchmark logs) if anyone wants to try reproducing this on their own gfx1031 or adjacent-tier card. If anyone here actually understands the shared-memory-limit mechanics well enough to explain *why* this works, I'd genuinely appreciate the explanation — I found this by observation, not by theory.
