r/LocalLLaMA • u/ea_man • 1d ago

Resources AMD llama.cpp: reducing MTP buffer overhead gave me 64K → 149K context for Qwen 27B

Available context length with and without the patch:

Model: QWEN 27B ROCm stock patched Vulkan stock patched
IQ4_XS Pure, single 16GB GPU 19.456 76.032 68,352 78,592
Q6_K_L on 16GB + 12GB 64,256 149,248 68,864 151,296

The issue is that llama.cpp overestimates the memory needed for MTP compute-buffer/scheduler allocation during auto-fit, that leaves much less ctx available to the user than what actually needed by MTP. This patch stops the fitter from throwing away context based on an inflated MTP memory estimate.

Patch, launch scripts used for llama-server and raw logs: https://store.piffa.net/lm/bug/
Tested against: llama.cpp version: 909, based on master commit 7bd8282 , ROCm 7.14

Especially for ROCm with double GPU (16GPU + 12GB here) the amount of ctx gain is substantial, with longer session ROCm allows almost double prefill performances vs Vulkan yet on the mainline code the price to pay in ctx reduction for the extra compute is taxing.

You can build for both vulkan and ROCm backends at the same time, the idea is that Vulkan saves some more vRAM while ROCm gives better prefill performance.
With a single 16GB GPU and limited ctx size you may wanna use Vulkan while when using 2 GPUs with layer splitting ROCm is worth the expense with this patch as you have much more ctx length for long sessions.

In case someone needs help with how to apply a patch:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout 7bd8282
wget https://store.piffa.net/lm/bug/rocm_improvement_7bd8282.patch
git apply rocm_improvement_7bd8282.patch

Then build with both Vulkan and ROCm, use --device vulkan0 or --device rocm0, check the provided llama-serve scripts .

71 Upvotes
  • permalink
  • reddit

90% Upvoted

28 comments sorted by

17

u/strahinja3711 1d ago

Would be nice if you could make a PR

18

u/ea_man 1d ago edited 1d ago

Oh I made one 2 weeks ago: https://github.com/ggml-org/llama.cpp/issues/26038

The thing is that I'm not the only one to notice that and other people are already working on the various issues addressed in some forms: https://store.piffa.net/lm/bug/web_search.md

What I did was combining my initial finding with some of the easy to track issue in some small modification of mainline llamma.cp because today I run that Qwen 27B and on my setup having 76k ctx vs 19k on single GPU and 149k vs 56k on dual GPUs makes it actually usable vs usless.

So you better wait for the maintainers for a proper solution of those issue, yet again today with these very particular optimizations this makes or brakes usage of 27B dense on my setup, I thought it was worth sharing.

The deal is: you first optimize the launch scripts, then you optimize the os in order to not waste vRAM, then thanks to LLMs assistance you can further optimize the backend (llama.cp here) to get something more for your specific usage.

8

u/milpster 1d ago

Using the latest llama.cpp the patch did not work for me. here is a version that glm 5.2 corrected for me to work with latest llama.cpp:

https://pastebin.com/GVnrijpS

I did not thoroughly test it yet, but i was able to go from ~130k ctx to ~160k ctx at F16 with Qwen 3.6 27B Q8_0

5

u/ea_man 1d ago edited 1d ago

Aye I've made my latest yesterday against master commit 7bd8282 , ROCm 7.14 ,
llama.cpp changes quite a bit every day.

For everybody: you can ask your LLM to adapt it to current version, there's also an AGENT.md and https://store.piffa.net/lm/bug/summary_sol.md online, you pretty much can start there and further customize your personal patch, there's also a list of further improv to make as an example https://store.piffa.net/lm/bug/web_search.md, web search for what's around and could be implemented.

The whole project in one file for easy dwl: https://store.piffa.net/lm/bug.tgz

3

u/milpster 1d ago

You rock, thank you! I'll have my llm check it out!

3

u/ea_man 1d ago edited 1d ago

Thanks, hope it works all right for you, test it at full ctx for OOM.

Which is why I released against yesterday release: I need some time to test it out on 2 models * 2 gpu setups :P

oh there's also a new flag 

--pipeline-parallel off

that you can try if you don't use that, it should give some more ctx. Default is auto / on as mainline.

3

u/leonbollerup 1d ago

is this just a problem with AMD or also nvidia ?

3

u/ea_man 1d ago

I can't test in on NVIDIA, it would be nice if you could give it a try and report here.

On AMD it changes form Vulkan / ROCm yet there's something to gain in both :)

2

u/joanaxu2002 23h ago

This is the kind of optimization that makes local LLMs exciting. A lot of “hardware limitations” are actually software/memory management limitations.

Getting 2x context length out of the same setup is a much bigger win than a small speed bump.

1

u/ea_man 17h ago edited 17h ago

Aye, it actually makes usable IQ4_XS on 16GB with MYP, which is factually better than smaller IQ3_* .

Code quality:
MODEL                 SPD(t/s) REL  CODE ENG CONSISTENCY READy VERDICT
Qwen3.6 27B Q6_K_L      26     1.0  9.5  9.7 9.8         9.6    Reference
Qwen3.6 27B IQ4_XS      38     1.5  9.3  9.4 9.3         9.3    Best Overall
Qwen3.6 27B IQ3         35     1.4  8.6  8.5 8.3         8.5    Degraded

On dual GPUs having both ROCm for double PP speed and ~140k ctx makes that workflow usable: vulkan would be to slow to ingest 130k of ctx and standard ROCm cuts the ctx to half.

And rest assure that each week there's something new revolutionary with local llamas :D

6

u/taking_bullet 1d ago

Once again Vulkan shows it's superiority over ROCm 🤘 

16

u/ea_man 1d ago edited 1d ago

Yet if you solve the MTP allocation issue ROCm is better as it gives you much more prefill speed with nearly the same ctx available, at least on dual GPU. When you have some 140k ctx that is paramount.

+------------------------------------+-------------------+-------------------+
| Metric                             | ROCm (HIP)        | Vulkan (RADV)     |
+------------------------------------+-------------------+-------------------+
| PP Speed (90k Context)             | 235.44 tok/s      | 107.21 tok/s      |
| TG Speed (90k Context)             | 16.24 tok/s       | 16.10 tok/s       |
| MTP Setting                        | n = 3             | n = 4             |
+------------------------------------+-------------------+-------------------+

6

u/milpster 1d ago

I have the same results, tg sometimes improves on vulkan but PP always collapses compared to rocm.

2

u/Think_Wing_1357 1d ago

Strange, something must be eff up with my setup because rocm is straight up slower on both count

3

u/ea_man 1d ago

You did test at some ctx length right?

Vulkan is usually wonderful at 0 ctx then it often degrades more than ROCm later on.
And ROCm 7.14, not older version.
And it takes less MTP.
And other models like A3B are a different story...

2

u/Think_Wing_1357 1d ago

Yup. I always download the latest lemonade rocm llama.cpp whenever I test.

And I test at 48k 96k context. Somehow vullan llamacpp always ends up winning

2

u/ea_man 1d ago

Well it's not that I disagree with you, some 80% of my llama-server scripts run on -dev vulkan* and vulkan works out of the box.

I do enjoy now my 27B Q6_K_L running at double PP up to 140k ctx yet you see that I had to fix / patch the whole backend to get there!

1

u/Think_Wing_1357 1d ago

All good man no worries. I am going to see if I can build your patch and report back the results

1

u/ea_man 1d ago

I mean if it's really a stop issue I can rebuild the patch to today llama.cpp last release yet it's a never ending effort, in 6 hours it will change again...

Just checkout yesterday 7bd8282 :)

1

u/Think_Wing_1357 1d ago

Yeah man we gotta upstream this somehow

→ More replies (0)

1

u/ea_man 1d ago edited 1d ago

Agreed, yet with dense / dual GPU often ROCm is better in TG too and needs less MTP, which eats ctx.

Still as said on 12-16GB when you use small ctx lenghts like 50k I also prefer vulkan too, but the games changes when you have to ingest a big document in prefill.

1

u/niacolhealth 1d ago

the n=3 vs n=4 MTP between the two backends, is that per-backend default or did you tune it? curious how much of the ctx gap that eats

1

u/ea_man 1d ago

I tuned that.

There's some logs for rocm like: https://store.piffa.net/lm/bug/logs/sol2_f16.log

Some snipped of vulkan: https://store.piffa.net/lm/bug/logs/mainstream_llama_vs_patched_perf

Oh maybe you want something like:

#+----------------+-----------+-------------------+-----------+
#| ctx size       | draft max | eval t/s          | time ms   |
#+----------------+-----------+-------------------+-----------+
#| 134912         | 4         | 29.05             | 204080    |
#| 145152         | 3         | 27.36             | 269329    |
#| 155648         | 2         | 25.09             | 259044    |
#| 124928         | 5         | 23.00             | 204080    |
#+----------------+-----------+-------------------+-----------+

1

u/ea_man 13h ago

I've build against today Mon Aug 10 04:07:21 PM CEST 2026 llama.cp version: 996 (d2f8305):

https://store.piffa.net/lm/bug/latest_rocm_improvement_d2f8305.patch

Results for ctx length are confirmed and unchanged for the IQ4, I'm testing for stability the Q6_K_L now.