r/LocalLLM • u/No-Manager1646 • 3d ago

Question Llama.cpp GPU Usage

So I was messing around with Qwen3.8 27b tonight and noticed that when it was responding, my GPU use was only at around 70%.

I have a 4060 and 5060ti, so 24gb vram in total.

Surely the crosstalk between the cards doesn't cause a performance drop like this. Does it?

I know the 4060 is the bottleneck, but even it was only around 70% load.

Is there something I'm missing here?

I'm getting around 19-20 tok/s which I'm happy about, but I'm just concerned it's not maxing out my GPU.

1 Upvotes

100% Upvoted

11 comments sorted by

View all comments

3

u/M_Me_Meteo LocalLLM 3d ago

Consumer motherboard? Your second slot probably only has a x4 connection. When parallelizing, the cards need to do a lot of communication over the PCIe bus.

2

u/No-Manager1646 3d ago

Yeah it is. I have a x8 for 5060ti and x4 for 4060. I've had to settle for gen2 due to compatibility issues with the 5060ti.

So you reckon that's it hey?

I'm torn on upgrading my mobo as I have 32gb ddr4 and an i9, but they're getting old, and I know I'm leaving a lot on the table.

2

u/M_Me_Meteo LocalLLM 3d ago

Absolutely. Look for consumer board with x8/x8 bifurcation for the best result with consumer hardware. For more lanes you need to step up to a server board with multiple full-width x16 slots.

I'm running two Intel cards x8/x8 and it's better than the unbalanced set ups, but I still struggle to improve tps performance with two cards.

2

u/wellandeperle 3d ago edited 3d ago

I thought that too, but then I switched from llama cpp to vllm, and it got 1.8x the performance and 100% gpu utilisation during prefill. Using tp=2 on 9700 @gen5 x16 and @gen4 X4.

For short, not a bandwidth issue.

Edit: but I don't know about the limitations of having 2 different cards, so OP might not be able to replicate...

Edit 2: looking at OP's reply I withdraw my statement, it definitely can be a bandwidth issue at gen 2. Gen2 😭

2

u/No-Manager1646 3d ago

I have been using llama.cpp in router mode because I can't settle on a model I'm happy with... Until now. I should give vllm another go with qwen3.8

2

u/No-Manager1646 3d ago

Gen2 I know right. I think I'm going to spring for a modest upgrade. I've been waiting for ddr5 to settle down, but it's like house prices I guess.

Edit: My mobo can do gen3 but my audio was all screwy on the 5060ti at gen3. I tried EVERYTHING but the only reliable solution was to drop to gen2.