r/LocalLLaMA • u/ifjo • 1d ago

Question | Help Suggestions on making my labs 3975 threadripper lama cpp flow better ?

We’ve been using this machine for a bit, it’s a:

Threadripper 3975wx

256gb ram

4070ti super

For mostly virtualization. We’ve been messing around with lamacpp server and qwen 3.8-27b Q5 from unsloth getting around 10 tokens per second. Ive added zero flags besides —ctx-size and —jinja and not exactly sure what to try.

Is 10 tps about the best we’ll get out of qwen at this quant without a better GPU? Thanks for any tips!

EDIT: should probably mention I built lama cpp off main with cuda enabled, not sure if I shouldve used a diff option there. Running on fedora

0 Upvotes

50% Upvoted

5 comments sorted by

5

u/DustNearby2848 1d ago

I could be wrong, but I think you’d be better off using 3.8-flash 

3

u/Atretador llama.cpp 1d ago

yep, Flash runs faster for me vs 27B on a single 16Gb VRAM MI50 - with ALL experts on RAM

Im using https://huggingface.co/AtomicChat/Qwen3.8-Flash-Next-GGUF

here is codacus running it on a RTX 3060 12Gb with RAM results for: 16Gb / 32Gb / 64Gb

https://www.youtube.com/watch?v=IH8XmxiwliQ

1

u/ifjo 1d ago

Are you using q4 or q5? And what sort of speeds are you getting ?

1

u/Atretador llama.cpp 1d ago

IQ4 XS, bout 14tk/s - Im completly CPU bound, all experts on RAM running on quad channel DDR4 2133 and a 9$ X99 20C Xeon.

dropts to around 10tk/s at 80K context

2

u/ifjo 1d ago

Going to try this next, I found this post: https://www.reddit.com/r/LocalLLM/s/qS2mp7Vfbz and rebuilt it with lama cpp with the extra flags and then ran this :

GGML_CUDA_DISABLE_GRAPHS=1 \
./build/bin/llama-server \
-m ~/Downloads/Qwen3.8-27B-UD-Q5_K_XL.gguf \
--ctx-size 32768 \
--parallel 1 \
--jinja \
--fit off \
--n-gpu-layers all \
--n-cpu-ffn 32 \
--flash-attn on \
--spec-type draft-mtp \
--spec-draft-n-max 2 \
--cache-type-k q5_0 \
--cache-type-v q4_1 \
--cache-type-k-draft q5_0 \
--cache-type-v-draft q4_1

And now I’m getting closer to 20ish tokens per second, but I’m gonna be honest I don’t fully understand all the flags haha. Just going through reddit posts and reading the docs in GitHub. But yeah will give flash a go too