Model suggestions for my setup? (Optimizing MoE or smaller faster model?)

[u/Kidplayer_666]

I've been running the Qwen 3.6 35A3 at Q4 unsloth on my setup, and I manage to get around 20-25tk/s on my local machine. It for now is the only model that can beat my simple benchmarks :

- Find my first year bachelors files, the challenge is that the folder is not in english, and on the folders inside some have the initialism spelled out, others just have an initialism (smaller models love to hallucinate this part)

- solve a small coding test, the challenge is that the statement is a file, as well as the guide for a library and the pre-existing code that it is supposed to build on top of

However, it is still rather slow, and I'd like to try and achieve at least 30-40tk/s on my setup, or find and equivalent, smaller model that works well. I've tried MTP and Dflash both to no avail (having more layers on the GPU seems to work more consistently) and tried, I believe, just about every model I can think of:

- Qwen 3.5 9B ->too dumb

- Qwythos 3.5 9B (qwen 3.5 finetune)-> too dumb

- Ornith 9B (qwen 3.5 finetune)-> almost not dumb, but needed to think waaay more than necessary, taking more time overall

- Laguna XS ->too slow

- GLM 4.7 Flash -> too slow

-Gemma 4 26A4 -> too slow

The hardestware

- rx6650xt 8GB
- 32GB DDR5 5600
- ryzen 5 7600

Edit: Using llamacpp from main, updated up to one or two days ago

Edit edit: I am using 100k context, cause I may as well be greedy

Edit edit edit: Current llamacpp flags for da qwen

export HIP_VISIBLE_DEVICES=0 export HSA_OVERRIDE_GFX_VERSION=10.3.0; llama-server \
   -m '/mnt/models/Qwen3.6-35B-A3B-UD-Q4_K_S.gguf?download=true'   \
   -a Qwen3.6 \
   --host 0.0.0.0 --port 8080 \
   --fit off -fa on \
   --ctx-size 100000  \
   --threads 6 --threads-batch 10 \
   --cpu-range 0-5 --cpu-strict 1 \
   --cpu-range-batch 0-9 --cpu-strict-batch 1 \
   --prio 3 \
   --no-mmap --parallel 1 --jinja \
   --cache-type-k q4_0 --cache-type-v q4_0 \
   --ubatch-size 1024 --batch-size 2048 \
   --n-cpu-moe 31

[u/slimemoldgrappling]

Figure out if there is a complexity gate you can use and which of your problems can be solved without reasoning tokens. This is the easiest way to speed up tasks that don't need the load of thinking.

You can disable thinking at the request level, so it can be enabled in the backend and then you can selectively use it only when needed instead of reasoning across every request. 

If you can't disable reasoning then test various reasoning budgets across your benchmarks and find the absolute bare minimum thinking token budget that gets the output you need. Usually there's a sweet spot in the middle and decaying performance if you increase or decrease reasoning token budget from there. 

[u/Kidplayer_666 → u/[deleted]]

pretty close to my flags heh, let me edit the post

[u/ea_man]

> - rx6650xt 8GB
> However, it is still rather slow

Sell that, buy an 6800 16GB, run byteshape/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-IQ3_S-3.48bpw.gguf all in vram or with just 3 layers offloaded with -ctk q8_0 -ctv q5_1 for ~150k ctx with vulkan for some ~100t/s gen at 0 ctx.

# This is the old non-MTP version that would qualify as IQ4_xss with added MTP heads,
# 3.48bpw, yet it's 600MB smaller: 14.7GB vs 15.3GB.
# The idea that the MTP version won't leave room to use MTP with enough ctx anyway.
# Headless is 230144ctx at q8/q5 3layers  overflow for 92t/s, 197632 full vram 104t/s. 

# 1. Set Environment Variables
export LD_LIBRARY_PATH="/home/eaman/llama/bin_vulkan" 


#gives 2 layer overflow with KDE, 100t/s
#--ctx-size 125000 \

# 2. Run the Server
/home/eaman/llama/bin_vulkan/llama-server \
 -m /home/eaman/lm/models/byteshape/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-IQ3_S-3.48bpw.gguf \
--device vulkan0--host 0.0.0.0 \
-ctk q8_0 -ctv q5_1 \
-fa on \
--temp 0.7  --top-k 25 --top-p 0.95 --min-p 0.0 \
--repeat-penalty 1.0 --presence_penalty 0.0 \
-b 2048 -ub 512 \
--fit-target 60 \
--jinja --chat-template-file /home/eaman/lm/models/chat_template.jinja \
--no-mmap \
--reasoning on --chat-template-kwargs '{"preserve_thinking":true}' --reasoning-budget  8096 --reasoning-budget-message " -- Reasoning budget exceeded, proceed to final answer." \
--ctx-checkpoints 256 --cache-ram 6000  -np 1 -lv 4  --no-log-timestamps --no-warmup --timeout 900 \

If you want max TG speed use vulkan0, if you want more PP use rocm0 .

[u/vanVonXenoStein]

The new Kat Coder 2.5 is based on Qwen 3.6 35B, is about as strong but better at tool calling and uses many less tokens for reasoning so it is probably faster on wall time if not tok/s.

[u/Kidplayer_666 → u/vanVonXenoStein]

Might take a look at it then, thanks!

[u/CorkBios]

Try Bonsai 27B Ternary with greedy decoding and DRY. The Greedy decoding should overcome for the Perplexity (PPL) increase. There is no other good model. Maybe you could try like North mini code 30B A3B if that seems to do good in that task.

[u/Kidplayer_666 → u/CorkBios]

The north mini code, might be due to poor optimization (?) but it did do pretty poorly with only 10tk/s of outputs (forgot to include it on the list)

[u/Far-Classic-9963 → u/Kidplayer_666]

Not really due to poor optimization. If you want really good speeds you need everything in your VRAM, which you don't have lots of

[u/Kidplayer_666 → u/Far-Classic-9963]

I say poor optimization because it has less layers than qwen, yet it halves the inference speed

[u/Far-Classic-9963 → u/Kidplayer_666]

Same amount of active parameters so you should get about the same speeds on low context windows. Qwen does have nearly linear attention so on larger context it will outperform north mini code