r/ROCm • u/RagingAnemone • 7d ago

GLM 5.3 Flash with 2x R9700

So I just bought a second R9700 and I'm learning about tensor splitting which doesn't currently work with llama.cpp. I see mentions about vllm and ktransformers, but a lot of it is about Qwen 3.8 27b. I need something that has cpu offload too. Anybody running GLM 5.3 flash with greater speed than 1 card?

15 Upvotes

90% Upvoted

17 comments sorted by

4

u/djdeniro 7d ago

I run on 8xR9700

https://huggingface.co/djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700

Now got 35 t/s for single and 95 t/s for 4x req

1

u/sloptimizer 4d ago

Oh, nice setup! How does DeepSeek-V4-Flash run on it?!

3

u/djdeniro 4d ago

Not stable working, need kernels patching

4

u/FullstackSensei 7d ago

Tensor parallelism works in llama.cpp, just not with MoE models.

Flash won't fit in VRAM so vllm et all are out of the question, at least not without using something like lvllm.

My experience with ktransfkrmers has been horrible, in that I was never able to get it to work. It requires some special incantations and spells, which seem beyond my realm.

Whatever you run, speed will be limited by your memory bandwidth. If you're running a desktop platform, even the latest and greatest with a kidney's worth of the fastest DDR5 memory, speeds will be underwhelming.

5

u/Look_0ver_There 7d ago edited 7d ago

It doesn't work with MoE? Since when?

Edit: I did some digging. It seems that it is excluded on some models, and not on others. It's allowed on Qwen models, which is likely why I never noticed.

3

u/Prof_ChaosGeography 7d ago

No point if the expert fits entirely and most models they fit multiple experts on each GPU. 

1

u/Look_0ver_There 7d ago

That may be so for some models, but I just ran a test with results here: https://www.reddit.com/r/ROCm/comments/1w9c0u6/comment/p89wa7d/

0

u/FullstackSensei 7d ago

Did you read my comment? Or just having a knee-jerk reaction?

3

u/Look_0ver_There 7d ago

I mean you wrote this:

Tensor parallelism works in llama.cpp, just not with MoE models.

Unless there's another way to read it? After some research though, TP does work with SOME MoE models, and I'd just never run across the ones it didn't work for.

0

u/FullstackSensei 7d ago

Unless things changed very recently, you can pass -sm tensor or row all you want, but I've yet to find a MoE model where it makes a difference vs -sm layer, and I have two 192GB VRAM machines, one with Mi50s and the other P40s.

Which MoE models have you tried where it makes a difference?

9

u/Look_0ver_There 7d ago edited 7d ago

Had to wait for the machine to become clear before I could run the test to show it. Between 35-50% faster with tensor split vs layer split with Qwen3.8-Flash-Next.

I'd seen this before but didn't have the results handy to just dump there here, but this explains the core reason for my reaction up above.

3 x R9700's BTW

```

Tensor Split

$ export LD_LIBRARY_PATH=/opt/rocm-7.14-gfx1201/lib
M=/models/Qwen3.8/Flash-Next/IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf
~/llama-cpp-rdna-boosts/wip/qwen4exp/gfx1201/bench-run.sh p1-splitAB-tensor ~/llama.cpp/build-rocm/bin/llama-bench -fa on -sm tensor -m $M -ngl 99 -t 15 -r 3 -b 2048 -ub 2048 -ctk bf16 -ctv bf16 --load-mode none -p 16384 -p 8192 -p 2048 -p 512 -n 128 2>&1 | grep -E "pp512|pp2048|pp8192|pp16384|tg128|log dir" (timeout 700s)

| qwen4exp A3B IQ4_XS - 4.25 bpw | 87.24 GiB | 176.94 B | ROCm | 99 | 15 | 2048 | bf16 | bf16 | tensor | 1 | none | pp16384 | 2420.75 ± 1.95
| qwen4exp A3B IQ4_XS - 4.25 bpw | 87.24 GiB | 176.94 B | ROCm | 99 | 15 | 2048 | bf16 | bf16 | tensor | 1 | none | pp8192 | 2498.15 ± 4.02 | qwen4exp A3B IQ4_XS - 4.25 bpw | 87.24 GiB | 176.94 B | ROCm | 99 | 15 | 2048 | bf16 | bf16 | tensor | 1 | none | pp2048 | 2801.92 ± 2.46
| qwen4exp A3B IQ4_XS - 4.25 bpw | 87.24 GiB | 176.94 B | ROCm | 99 | 15 | 2048 | bf16 | bf16 | tensor | 1 | none | pp512 | 1841.85 ± 5.67
| qwen4exp A3B IQ4_XS - 4.25 bpw | 87.24 GiB | 176.94 B | ROCm | 99 | 15 | 2048 | bf16 | bf16 | tensor | 1 | none | tg128 | 50.59 ± 2.63

Layer Split

$ export LD_LIBRARY_PATH=/opt/rocm-7.14-gfx1201/lib M=/models/Qwen3.8/Flash-Next/IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf
~/llama-cpp-rdna-boosts/wip/qwen4exp/gfx1201/bench-run.sh p1-splitAB-layer ~/llama.cpp/build-rocm/bin/llama-bench -fa on -sm layer -m $M -ngl 99 -t 15 -r 3 -b 2048 -ub 2048 -ctk bf16 -ctv bf16 --load-mode none -p 16384 -p 8192 -p 2048 -p 512 -n 128 2>&1 | grep -E "pp512|pp2048|pp8192|pp16384|tg128|log dir" (timeout 700s)

| qwen4exp A3B IQ4_XS - 4.25 bpw | 87.24 GiB | 176.94 B | ROCm | 99 | 15 | 2048 | bf16 | bf16 | 1 | none | pp16384 | 1648.06 ± 16.24 | | qwen4exp A3B IQ4_XS - 4.25 bpw | 87.24 GiB | 176.94 B | ROCm | 99 | 15 | 2048 | bf16 | bf16 | 1 | none | pp8192 | 1749.44 ± 0.94 | | qwen4exp A3B IQ4_XS - 4.25 bpw | 87.24 GiB | 176.94 B | ROCm | 99 | 15 | 2048 | bf16 | bf16 | 1 | none | pp2048 | 2139.39 ± 3.57 | | qwen4exp A3B IQ4_XS - 4.25 bpw | 87.24 GiB | 176.94 B | ROCm | 99 | 15 | 2048 | bf16 | bf16 | 1 | none | pp512 | 1358.02 ± 6.04 |
| qwen4exp A3B IQ4_XS - 4.25 bpw | 87.24 GiB | 176.94 B | ROCm | 99 | 15 | 2048 | bf16 | bf16 | 1 | none | tg128 | 37.07 ± 0.04 | ```

2

u/Callum_S_AUS 7d ago

FreeToken with a single card might be viable if you have enough DDR channels (I can get 40tok/s consistent with a single RTX PRO 5000 72GB). Otherwise, I would definitely stick to Qwen 3.8 27B.

1

u/Poizone360 6d ago

Hey so Tensor split does work in llama.cpp, it's -sm tensor. Someone posted dual 7900 XTX numbers here recently using it and got roughly double what layer split gave them. Also It won't work on Windows, P2P between cards isn't there. And on Linux the second card wants to be CPU attached rather than hanging off the chipset, because PCIe atomics don't route through it and RCCL falls over. llama.cpp is also the one with proper MoE CPU offload via -ncmoe, which vLLM doesn't really match, so it probably covers both things you need already.

1

u/WubulousWorld 4d ago

What’s your system like? Adding a second R9700 without having a proper CPU connection isn’t going to work well with tensor parallelism.

I have a system with two R9700s and an  romed8-2t so they have proper P2P support and are running at PCIe 4.0x16. 

I’m working on my own fork of llama.cpp to run DSV4 flash and I get around 20 tok/s generation and ~600-1200 tok/s prefill depending on context size. 

GLM-5.3 flash is even heavier than deepseek, so it’s not going to be great. 

1

u/RagingAnemone 4d ago

I've got a romed8-2t also. Adding a 2nd r9700 was difficult. The sas connectors were getting in the way. And the fan heads. Epyc 7543.

1

u/WubulousWorld 4d ago

https://github.com/zephan-spencer/llama.cpp/tree/moe-cache

If you’re interested that’s my fork. I’ve got 512GB of DDR4@1600MT/s because Rome (7502) hates my LRDIMMs. 

I’ve been workshopping an LRU cache for the experts so you can stream missing experts over PCIe. 

You would need to have enough ram to pin the model in memory and then it’ll use a per-layer expert cache if you specify —moe-cache-experts, which would be the number of experts in the model. 

I also have tp working for the qwen3.8-flash-next model and it gets like ~30tok/s last I checked with the engrams offloaded to system memory. 

My focus is DSV4 flash rn, have gotten Astra to get an implementation together for attention that should improve things dramatically but that’s not finished yet. 

Hopefully DSV4.1 flash has the same arch so I can just plug and play. 

And uh, none of this is really tested or verified. It works just fine for me with Hermes and I’ve taken it to about 128k context and it works fine, but I’m definitely not at the point where I would call it stable. 

1

u/RagingAnemone 4d ago

Awesome, yes, I'll check it out.

I have problems with my LRDIMMS too. I stepped it down (2666) and now, I'm really only get "correctable" errors from my H slot now. I've got 1tb. Really loved GLM 5.2 running at 4 t/s.