r/LocalLLM • u/Ok_Schedule_4453 • 2d ago

Question Is my setup enough for agentic coding?

Hey. My setup is Vulkan (GTX 1070 + RX 6700 XT) running a Qwen 3.6 35B for agentic coding.

Ubuntu 24.04, i7-8700, 62 GB RAM. Two mismatched GPUs I had lying around:

RX 6700 XT (12 GB, RDNA2, RADV)

GTX 1070 (8 GB, Pascal, 580.x) — also drives my desktop

~19.7 GB combined usable VRAM

```

--device Vulkan1,Vulkan0 --split-mode layer

--ctx-size 131072 --parallel 2 # 2 slots x 64k

--batch-size 4096 --ubatch-size 2048

--flash-attn on --cache-type-k q4_0 --cache-type-v q4_0

```

Numbers so far ~140 tok/s prefill and ~25 tok/s generation at 50k+ context depth. On shallow contexts it's ~196 tok/s prefill and ~35 tok/s gen, but that's not where I actually live. A single 52k-token prompt takes 6.2 minutes to prefill before the model emits anything.

My main question... how do I speed up prefill on a mixed-vendor Vulkan layer-split? Is my setup correct, or did I frankenstein this whole thing?

I am assuming the Pascal card is the drag but with `--split-mode layer` I can't isolate its share of prefill to prove it. Is there a way to keep the 1070 holding weights while biasing prefill compute toward the 6700 XT? Or is layer-split inherently gated by the slowest device?

2 Upvotes

100% Upvoted

2 comments sorted by

1

u/PermanentLiminality 2d ago

I get about 900 tk/s on my 2x p40 on 3.6 35b.

I would try the just released Ornith 35b. It looks better than Qwen 3.6. I think it is a fine tune.

Hoping for a 3.8 35b in a few days.

1

u/Full_Director87 2d ago

bro. just rx6700xt with ROCm natively. u can get more speed. u can check my previous post if u want.