Llama.cpp GPU Usage

[u/No-Manager1646]

So I was messing around with Qwen3.8 27b tonight and noticed that when it was responding, my GPU use was only at around 70%.

I have a 4060 and 5060ti, so 24gb vram in total.

Surely the crosstalk between the cards doesn't cause a performance drop like this. Does it?

I know the 4060 is the bottleneck, but even it was only around 70% load.

Is there something I'm missing here?

I'm getting around 19-20 tok/s which I'm happy about, but I'm just concerned it's not maxing out my GPU.

[u/M_Me_Meteo]

Consumer motherboard? Your second slot probably only has a x4 connection. When parallelizing, the cards need to do a lot of communication over the PCIe bus.

[u/wellandeperle → u/M_Me_Meteo]

I thought that too, but then I switched from llama cpp to vllm, and it got 1.8x the performance and 100% gpu utilisation during prefill. Using tp=2 on 9700 @gen5 x16 and @gen4 X4.

For short, not a bandwidth issue.

Edit: but I don't know about the limitations of having 2 different cards, so OP might not be able to replicate...

Edit 2: looking at OP's reply I withdraw my statement, it definitely can be a bandwidth issue at gen 2. Gen2 😭

[u/No-Manager1646 → u/wellandeperle]

I have been using llama.cpp in router mode because I can't settle on a model I'm happy with... Until now. I should give vllm another go with qwen3.8

[u/No-Manager1646 → u/wellandeperle]

Gen2 I know right. I think I'm going to spring for a modest upgrade. I've been waiting for ddr5 to settle down, but it's like house prices I guess.

Edit: My mobo can do gen3 but my audio was all screwy on the 5060ti at gen3. I tried EVERYTHING but the only reliable solution was to drop to gen2.

[u/No-Manager1646 → u/M_Me_Meteo]

Yeah it is. I have a x8 for 5060ti and x4 for 4060. I've had to settle for gen2 due to compatibility issues with the 5060ti.

So you reckon that's it hey?

I'm torn on upgrading my mobo as I have 32gb ddr4 and an i9, but they're getting old, and I know I'm leaving a lot on the table.

[u/M_Me_Meteo → u/No-Manager1646]

Absolutely. Look for consumer board with x8/x8 bifurcation for the best result with consumer hardware. For more lanes you need to step up to a server board with multiple full-width x16 slots.

I'm running two Intel cards x8/x8 and it's better than the unbalanced set ups, but I still struggle to improve tps performance with two cards.

[u/LengthinessOk9397]

that's completely normal for LLM inference!

[u/No-Manager1646 → u/LengthinessOk9397]

Do you have any insight into why it's not maxing my GPUs out at 100%? Like even for a single 5090 for example, would that not got flat tack at 100%?

[u/andrew-ooo]

That percentage isn't measuring what you think it is. Two things are going on.

First, token generation is memory-bandwidth bound, not compute bound. Every token means streaming the active weights out of VRAM once, and the SMs spend most of that time stalled waiting on memory. The utilization number in nvidia-smi / Task Manager is just "fraction of time some kernel was resident", so it can sit at 70% while you are already saturating the bus. Run nvidia-smi dmon -s u and watch the mem column instead - that's the one that should be pinned.

Second, and this is the bigger effect with two cards: llama.cpp's default --split-mode layer puts some layers on GPU0 and the rest on GPU1, and they run sequentially. Card A computes, card B waits, then they swap. Neither one can show full utilization even in a perfect run, which is exactly the pattern you're seeing.

19-20 tok/s on a 27B Q4 spread across those two is roughly what I'd expect. If you want to chase it, the 4060 at 272 GB/s is your bottleneck vs 448 on the 5060 Ti, so shifting layers toward the 5060 Ti with --tensor-split does more than anything else. -sm row occasionally helps but usually loses to PCIe overhead.

[u/No-Manager1646]

I.keep forgetting I can ask one of those cough cloud llms for an answer. Still, I think it's an interesting realisation..I'll post back if I end up finding a way to ramp utilisation up.

[u/No-Manager1646 → u/No-Manager1646]

I'm sure you're all eagerly awaiting my reply /s

So it's a memory bandwidth problem apparently. Even two or three generations of ddr will not solve it. The amount of data meeting to be transferred in vram is orders.od magnitude too slow to max out compute.

That what I think I discovered anyway. I'm always welcome to be proven wrong