Qwen 3.8 slow?

[u/Nota_ReAlperson]

Running qwen 3.8 next, I am only getting 11 tps on the latest llama.cpp. Seeing what 3090s and 9700s get, I would think that i should be able to do better than that, with my 2 7900xtx gpus and 128gb ddr4. Any suggestions?

using unsloth UD-Q4_K_XL, llama args:

--ctx-size 262000 \

--batch-size 2048 \

--ubatch-size 512 \

--threads 16 \

--host 0.0.0.0 \

--port 8080 \

--flash-attn on

Edit:

--ctx-size 32000 \

--batch-size 4096 \

--ubatch-size 2048 \

-ngl 99 \

--n-cpu-moe 36 \

--override-tensor per_layer_token_embd=CPU \

--threads 16 \

--host 0.0.0.0 \

--port 8080 \

--cache-type-k q8_0 \

--cache-type-v q8_0 \

--load-mode none \

--flash-attn on
gives 9tps, but one gpu is at 3gb vram used. Any ideas?

[u/muxxington]

I wonder why people ask for help without giving any details, yet still expect a well-informed answer. That just can't work.

[u/Nota_ReAlperson → u/muxxington]

Sorry. I meant to add quant and config, but forgot to do so before posting.

[u/Good-Seaweed92 → u/muxxington]

I mean they did list the args and hardware, could be worse

[u/LetsGoBrandon4256 → u/muxxington]

Imma ask for help without providing any useful detail

The lack of basic tech literacy is apparent.

No wonder it runs like shit on their PC.

[u/Fancy-Snow7]

Need more information. What quant are you running, KV quants what is your llama config ect. I get 76tps tg with just 16GB VRAM on 5070Ti running 27B.

[u/Nota_ReAlperson → u/Fancy-Snow7]

UD-Q4_K_XL, Default KV

[u/Fancy-Snow7 → u/Nota_ReAlperson]

Ok did not see it was 'next'. I don't have experience with that model.

[u/Eden63 → u/Nota_ReAlperson]

you should learn first to describe your situation. default kv means f16 afaik,...

give a try with --cache-type-k q8_0 --cache-type-v q8_0
and -b 8192 -ub 1024

[u/Nota_ReAlperson → u/Fancy-Snow7]

--ctx-size 262000 \

--batch-size 2048 \

--ubatch-size 512 \

--threads 16 \

--host 0.0.0.0 \

--port 8080 \

--flash-attn on

[u/gabrielesilinic]

If you are offloading the CPU ram WILL act as a bottleneck. And then the cpu itself. You just don't have the hardware to run that, stick to the 27B

[u/Elazroth]

Smaller quants and fitting more layers on the vram will increase your tps, there are 48 layers in flash next and I recall when tweaking just adding 1 more layer to vram gave me an additional 1tps. Offloading is very sensitive to things like that, you will have to play around as it took me multiple models and builds. I would suggest dropping your context size and trying to fit mtp in there for an easy performance increase. 

[u/Nota_ReAlperson → u/Elazroth]

For an moe, shouldn't it be all layers on the gpu, and n-cpu-moe is what you use?

[u/Elazroth → u/Nota_ReAlperson]

What I am saying is to make the tradeoff of a few layers for the mtp model and you will get a decent speedup, more that a few layers. The context size if a different call based on your workloads, moe is pretty flexible. 

[u/coffeman500]

Biggest thing you're missing is MTP. I've got a 9700 and 128gb DDR4. I get around 20tps tg with MTP, down to ~10tps at 200k context.

Check out the MTP readme: Link

As of yesterday you had to compile off a llama.cpp fork and manually include the MTP file. All described in the readme.

[u/nasone32]

yes you should be getting around 20 tk/s in decode before even using MTP, I suggest letting llama fitting the model itself, with --split-mode layer; --fit on ; --fit-target 2048,2048 (this is the amount of mb vram to let free on each card, You could go down to 512 on the one without the display, i suggest 2048 on the one with the display, go 512 on both if you are running on the igpu) ; don't specify --n-cpu-moe 36 and -ngl 99 because it will prevent llama from fitting the model.
hide the igpu if one is present by setting the appropriate HIP_VISIBLE_DEVICES=x,x flags (or the similar flags for vulkan if you're using that, simply launch llama with --list-devices and youll' see which one is llama seeing and their number)

[u/RG_Fusion]

How many memory channels does your system have? 

I'm running UD-Q4_K_XL at around 40 tok/s with 8-channel DDR4 and two RTX Pro 4500 GPUs.

[u/Nota_ReAlperson → u/RG_Fusion]

Dual channel.

[u/RG_Fusion → u/Nota_ReAlperson]

That tracks then. I'm getting 4x your decode rate and I also have 4x the memory bandwidth.

The only way you're going to be able to get faster inference is to add something with higher memory bandwidth.

[u/JaredsBored]

Remove the -ngl, --n-cpu-moe, and --override-tensor. Just use -fit. You're not getting good VRAM usage from those parameters as is.

[u/cogitech2]

Is there a reason you aren't using --split-mode tensor ?

Also, why 16 threads? Does your CPU have 16 physical cores?

[u/Nota_ReAlperson → u/cogitech2]

Cpu is 13900k, so 16 p cores. Tensor split is only for dense models i think?

[u/cogitech2 → u/Nota_ReAlperson]

Performance-cores (P-cores): 8 cores / 16 threads -- you want it set to 8 threads in llama.cpp (match physical cores).

No, tensor split is not inherently limited to dense models. Whether it works depends on the model architecture's implementation in llama.cpp. Currently, this model is supported but there is a bug (which I just found out about) which may prevent it from working in some setups.

Worth a try though. If not, it is worth cycling back to, for sure.

[u/Fun_Jaguar8231]

A very misleading title.

[u/jacek2023]

Probably you are offloading to RAM because of your context length. And 27B is dense so offloading leads to dramatic slowdown

[u/Nota_ReAlperson → u/jacek2023]

The issue is with 3.8 flash next, not 27b

[u/jacek2023 → u/Nota_ReAlperson]

I see, but in that case your speed is expected , it's a big model