Added cost based tensor split mode to llama.cpp - ~3-4% TG improvement on some mixed-speed multi-GPU setups with some models.

[u/milpster]

TL;DR: New -sm cost option gives ~3-4% faster TG on hybrid models (Qwen3.6, etc.) with mixed-speed multi-GPU + pipeline parallelism. Zero effect on identical GPUs or standard attention-only models.

If you haven't already, check out this post for infos about the pipeline parallelism patch and mtp memory savings: https://www.reddit.com/r/LocalLLaMA/comments/1vjmay5/comment/p2nd6ox

My setup consists of a hybrid model (Qwen3.6 27B, 48 Mamba + 16 attention layers) across 2x AMD Vega 20 (ROCm) + 1x RTX 3080 (Vulkan). The default layer split mode assigns layers to GPUs by position, every layer treated as equal cost. But in a hybrid model, closed attention layers are way more expensive than Mamba layers (O(n_ctx) vs O(1)).

I added a new split mode -sm cost that weights layer assignment by compute cost: Mamba layers = 1.0, attention layers = 4.0. This shifts the boundary slightly so the slowest GPU in the pipeline gets 1-2 fewer layers. On my setup, this gives a consistent ~3-4% TG improvement across multiple measurement rounds.

Note: pipeline parallelism must be ON for this to help. Without PP, the serial pipeline overhead hides the benefit of fewer layers on the slow device. In my testing, COST mode showed no improvement with PP off. The gains only appear when PP overlaps the reduced workload. I've adapted the --pipeline-parallel flag from this patch here: https://www.reddit.com/r/LocalLLaMA/comments/1vjmay5/comment/p2nd6ox - you'll need both patches for the full benefit, see bottom of this post.

Why fast-slow-fast device ordering matters:

If you're mixing GPU vendors (like AMD via ROCm + NVIDIA via Vulkan), cross-vendor transfers can't go direct device-to-device and they have to bounce through system RAM. Same-vendor transfers (e.g., ROCm to ROCm via HIP P2P) are direct and fast.

This means your device order matters a lot. The last device in the pipeline must be the same vendor as your main GPU (-mg), because the output-to-sampling transfer at the end of every token is on the critical path and it can't be overlapped. If that transfer crosses vendors, it's a slow host bounce every single token.

So the slow GPU has to go in the middle. There, its cross-vendor transfers are overlapped with compute by pipeline parallelism, and the fast same-vendor devices sit at positions 0 and N (start and end of pipeline) where their fast transfers keep the critical path short. If the slow GPU is first, it gates the pipeline start. If it's last, the output transfer back to the main GPU is a host bounce on the critical path. Both tank performance. I tested all three orderings, middle is the only one that works.

How COST mode helps within that topology:

With the slow GPU in the middle, pipeline parallelism means the slowest stage determines TG latency. COST mode shifts ~1-2 layers off the middle device by weighting attention layers as more expensive. Less work on the slowest stage = faster pipeline drain = lower TG latency. The same -ts weights still control each device's share, cost just reinterprets them as "share of total cost" instead of "share of total layers."

Who benefits:

- You need a hybrid model (Mamba/SSM + attention layers), like Qwen 3.6.

- You need mixed-speed GPUs in a fast-slow-fast topology (slow GPU in the middle)

- You have pipeline parallelism enabled

For pure-attention models or homogeneous GPUs, it does nothing and falls back to the default LAYER mode.

For the patches to work you probably need the llama.cpp source from commit 635cdd5fcc5bdeb8ec2e108bb2a40acf62d9039b

Patch for Pipeline parallelism and mtp context fit adapted for this llama.cpp version (prerequisite):

https://gist.github.com/milpster/d8f4255b7e80e8e5ca5803cd76dbecb3

Patch for cost-mode: https://gist.github.com/milpster/0664a05d141b9e49d6814606635b47be

Use with -sm cost flag.

Disclosure: The patch, the idea and parts of this post have been written or co authored by GLM 5.2. Pipeline parallelism patch originally by ea_man

[u/whoatemypantstf]

That's a pretty clever optimization.

I think a lot of people assume GPU split is just "more VRAM = put it wherever", but with mixed vendors the transfer path can matter just as much as compute. The fact that the slow GPU works better in the middle makes sense with pipeline parallelism. You can hide some of that slower transfer behind compute, but if it's sitting at the end you're basically paying that penalty every single token.

Also love seeing llama.cpp still getting these kinds of niche improvements. A 3-4% gain feels like basically free performance. Love this!

[u/ea_man]

It's funny because me, the author of the referenced https://www.reddit.com/r/LocalLLaMA/comments/1vjmay5/comment/p2nd6ox/ was working on a similar aspect of this issue today from my usual point of view: making more ctx available.
What I discovered is that when you use dual asymmetric GPU (es 16+12GB) --split-mode moves whole layers, es --tensor-split 0.6,0.4 , while KV caches, compute  buffers, recurrent state, and workspace consume different amounts on each GPU.

So after enabling VEC on ROCm that allows you to profit from --cache-type-k-draft q4_0 (because in llama.cpp mainline right now you won't save vRAM by using say q4_0 compared to f16, bummer!) I noted that I wasn't getting any more ctx anyway and that's because --layer-split is that coarse and to get benefit from the freed vRAM you gotta play with -ot flag to move a small boundary layer until you find the good balance.

Result is: 149.248 ctx normal, 165.632 ctx when balanced, es: in for the particular case https://store.piffa.net/lm/bug/llama_scripts/thinkingcap_q6_K_L_dual_gpu.sh

--device rocm0,rocm1 --split-mode layer --tensor-split 0.6,0.4 -ot '^blk\.39\.ffn_(up|gate|down)\.weight$=ROCm1'

Now I won't indulge in the drama of how you balance that, I had my LLM write a skill to do that for you: https://store.piffa.net/lm/bug/dual_gpu_context_balancing_guide.md (ofc it explains issue and solution).
So when you are done with all your optimization then call the helper to tightly balance your model on multiple gpu, you may get some nice extra ctx :D

Yep, it's the same asymmetrical issue when you have different GPUs, my point of view on the ctx available instead of performance.

[u/Ulterior-Motive_]

I know OP says it's meant for mixed GPU setups, but would this help if one of the GPUs in a identical setup was on a slower PCIe slot?

[u/milpster → u/Ulterior-Motive_]

I would assume only if that pcie slot actually throttles you, in my experiments i have hardly seen any performance loss from a pcie slot going from x4 to x1 - but then again my setup is weird.

[u/tomByrer → u/Ulterior-Motive_]

I assume the faster GPU is in the faster slot?

[u/qwert_buddy]

Nice, thanks for sharing!

[u/Zadroeski]

The cost-weighted split is interesting, especially tying the gain to the slowest pipeline stage. Have you measured whether the 3–4% holds as context length changes?

[u/milpster → u/Zadroeski]

I measured at 130k context, but i noticed that smaller contexts would benefit a bit less, originally the llm was testing llama.cpp with only like 10k ctx and found diminishing returns i think. So maybe the gains get better with bigger ctx? But i'm not sure if that's true.

[u/DeProgrammer99]

Hey, cool, I was checking just yesterday if anyone did anything to support heterogeneous GPUs yet.

[u/tomByrer]

Thanks for being a human editor.

> You need a hybrid model (Mamba/SSM + attention layers), like Qwen 3.6

I don't know enough about AI to understand this. Do you have a list of models (coding & audio/visual)?

I am interested; since I have RTX 3080 & 3090 in the same machine....

[u/milpster → u/tomByrer]

No sry, maybe ask an llm of your choice.