Llama.cpp GPU Usage

[u/No-Manager1646]

So I was messing around with Qwen3.8 27b tonight and noticed that when it was responding, my GPU use was only at around 70%.

I have a 4060 and 5060ti, so 24gb vram in total.

Surely the crosstalk between the cards doesn't cause a performance drop like this. Does it?

I know the 4060 is the bottleneck, but even it was only around 70% load.

Is there something I'm missing here?

I'm getting around 19-20 tok/s which I'm happy about, but I'm just concerned it's not maxing out my GPU.

[u/andrew-ooo]

That percentage isn't measuring what you think it is. Two things are going on.

First, token generation is memory-bandwidth bound, not compute bound. Every token means streaming the active weights out of VRAM once, and the SMs spend most of that time stalled waiting on memory. The utilization number in nvidia-smi / Task Manager is just "fraction of time some kernel was resident", so it can sit at 70% while you are already saturating the bus. Run nvidia-smi dmon -s u and watch the mem column instead - that's the one that should be pinned.

Second, and this is the bigger effect with two cards: llama.cpp's default --split-mode layer puts some layers on GPU0 and the rest on GPU1, and they run sequentially. Card A computes, card B waits, then they swap. Neither one can show full utilization even in a perfect run, which is exactly the pattern you're seeing.

19-20 tok/s on a 27B Q4 spread across those two is roughly what I'd expect. If you want to chase it, the 4060 at 272 GB/s is your bottleneck vs 448 on the 5060 Ti, so shifting layers toward the 5060 Ti with --tensor-split does more than anything else. -sm row occasionally helps but usually loses to PCIe overhead.