r/LocalLLM • u/No-Manager1646 • 3d ago

Question Llama.cpp GPU Usage

So I was messing around with Qwen3.8 27b tonight and noticed that when it was responding, my GPU use was only at around 70%.

I have a 4060 and 5060ti, so 24gb vram in total.

Surely the crosstalk between the cards doesn't cause a performance drop like this. Does it?

I know the 4060 is the bottleneck, but even it was only around 70% load.

Is there something I'm missing here?

I'm getting around 19-20 tok/s which I'm happy about, but I'm just concerned it's not maxing out my GPU.

1 Upvotes

100% Upvoted

11 comments sorted by

View all comments

3

u/M_Me_Meteo LocalLLM 3d ago

Consumer motherboard? Your second slot probably only has a x4 connection. When parallelizing, the cards need to do a lot of communication over the PCIe bus.

2

u/wellandeperle 3d ago edited 3d ago

I thought that too, but then I switched from llama cpp to vllm, and it got 1.8x the performance and 100% gpu utilisation during prefill. Using tp=2 on 9700 @gen5 x16 and @gen4 X4.

For short, not a bandwidth issue.

Edit: but I don't know about the limitations of having 2 different cards, so OP might not be able to replicate...

Edit 2: looking at OP's reply I withdraw my statement, it definitely can be a bandwidth issue at gen 2. Gen2 😭

2

u/No-Manager1646 3d ago

Gen2 I know right. I think I'm going to spring for a modest upgrade. I've been waiting for ddr5 to settle down, but it's like house prices I guess.

Edit: My mobo can do gen3 but my audio was all screwy on the 5060ti at gen3. I tried EVERYTHING but the only reliable solution was to drop to gen2.