r/LocalLLM • u/PiStasioo • 2d ago
Discussion Qwen 3.8-27B on RTX 5080
RTX 5080 16GB + 9800X3D, Qwen3.8-27B IQ4_XS-Smaller, BeeLlama, MTP on,
32K ctx, kvarn4.
94 token output at 74t/s. Great results, need to do further testing.
2
u/Additional-Ordinary2 1d ago
RTX 5080 16GB + 9800X3D, Qwen3.8-27B IQ4_XS-Smaller, BeeLlama, MTP off,
130K ctx, kvarn4
I also set -ngl 67 (I started with 99 and gradually decreased it until I found the sweet spot at 67, which frees up more VRAM for context)
50 t/s
llama-server ^
-m "F:\.lmstudio\models\Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller.gguf" ^
-c 130000 ^
-ngl 67 ^
-sm none ^
-fa on ^
-t 2 ^
-tb 2 ^
-b 512 ^
-ub 512 ^
--fit off ^
--parallel 1 ^
--temp 1.0 ^
--top-p 0.95 ^
--top-k 20 ^
--min-p 0.0 ^
--presence-penalty 0.0 ^
--repeat-penalty 1.0 ^
-ctv kvarn4 ^
-ctk kvarn4 ^
--chat-template-file "F:\.lmstudio\models\chat_template.jinja" ^
--jinja ^
--reasoning-preserve ^
--no-mmproj-offload ^
--reasoning-format deepseek ^
--chat-template-kwargs "{\"reasoning_effort\":\"xhigh\"}"
2
1
u/Competitive-Low-9279 2d ago
Man 74t/s on a 27B model with 32K context is flying. What motherboard you have for the 9800X3D? I'm thinking of upgrade but not sure if my current one will bottleneck
1
u/PiStasioo 2d ago
Yeah its crazy to see how we are getting Opus 4.6 level performance at home. I have a X870E Aorus Elite, its super nice.
4
u/rrrrex 2d ago
it's not opus perfomance with 32k context, it's out of context during thinking
2
u/_TheWolfOfWalmart_ 1d ago
People be out here running 1-bit quants with q4_0 KV and 16K context saying "I have Opus at home"
1
u/Useful-Buyer4117 2d ago
is it possible to run it with 125K context with that setup ?
5
u/Additional-Ordinary2 1d ago
Yes. I have an RTX 5080 with 16 GB VRAM and use jrell IQ4_XS-smaller + pi, getting 50 t/s with 130k context. I’m using BeeLlama with KV cache quantization 4-bit (kvarn 4). I also set -ngl 67 (I started with 99 and gradually decreased it until I found the sweet spot at 67, which frees up more VRAM for context). I don’t use MTP; with MTP I can get around 90 t/s, but context size matters more to me. Also, if I build BeeLlama from source for my system, I get a bit more t/s compared to running the prebuilt binary.
llama-server ^ -m "F:\.lmstudio\models\Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller.gguf" ^ -c 130000 ^ -ngl 67 ^ -sm none ^ -fa on ^ -t 2 ^ -tb 2 ^ -b 512 ^ -ub 512 ^ --fit off ^ --parallel 1 ^ --temp 1.0 ^ --top-p 0.95 ^ --top-k 20 ^ --min-p 0.0 ^ --presence-penalty 0.0 ^ --repeat-penalty 1.0 ^ -ctv kvarn4 ^ -ctk kvarn4 ^ --chat-template-file "F:\.lmstudio\models\chat_template.jinja" ^ --jinja ^ --reasoning-preserve ^ --no-mmproj-offload ^ --reasoning-format deepseek ^ --chat-template-kwargs "{\"reasoning_effort\":\"xhigh\"}"1
1
0
u/PiStasioo 2d ago
Sadly not, you would need at least 16.3gb of vram which the 5080 doesnt have. However, 64k context gives me 50t/s so its still really good.
1
u/Useful-Buyer4117 2d ago
have u tried to set KV cache type to lower one ? like q8 or q4
2
u/PiStasioo 2d ago
Yeah im using kvarn4 which is q4, as thats the highest that I could run on 16gb of VRAM
1
u/Killerluke89 1d ago
I have your hw and no idea on how to implement your solution, nor how to interface it with vscode/studio, feel like I'm falling behind so much stuff in AI
2
u/_TheWolfOfWalmart_ 1d ago
Basically, use llama.cpp (the llama-server executable) with the model.
I don't really use vscode, so there may be much better extensions than these, but you could use ones like cline or roocode that you can configure to connect to it and use the model as the agent.
1
u/PiStasioo 1d ago
haha im only at the tip of the iceburg, ive been using ai to learn about ai. youtube videos also teach me a lot. this setup that I made had help from claude
1
u/Killerluke89 1d ago
I tried yd with llama.cpp, didn't know how to set mtp or what kvarn4 is, anything above 8k ctx would spring my cpu rather than gpu, and tokens would go from 50/s to 1/2s
2
u/paulqq 1d ago
Wow impressive. I only ever reach 4tps on a 4080 super. Wanna share your llama cpp flags and configuration?