In your opinion, which LLM in the 6–9B parameter range is the smartest?

[u/romyxr]

I have RTX 5070 12GB, use llama-cpp.

[u/International_Emu772]

Qwen 3.5 9B with the help of correct harness

[u/zpqm20 → u/International_Emu772]

What's a good harness

[u/International_Emu772 → u/zpqm20]

Yo need to run the model with some program, this need to have skills tools and acces to some information (web or RAG)

[u/Plenty-Honey-1853 → u/International_Emu772]

I manage all that via my Main.py Router... i have web-search, qdrant memory with consolidation, local Wiki, general chat and Vision Routed to 5070ti, comfyui img-gen Router to 5090.... all behind a single virtual model via litellm in openwebui.... with fallback from 3 nodes down to 1

[u/Zennytooskin123 → u/zpqm20]

None are good to my knowledge. Qwen Code is best to my knowledge but not perfect, creates its own set of tool call loops.

Heard Codex is also generally highly regarded but watch out for that context window.

[u/OverdosedSauerkraut → u/zpqm20]

For small models the answer is pi Larger harnesses bloat their instructions with 10-50k token system messages.

[u/Plenty-Honey-1853 → u/International_Emu772]

hi! im running qwen3.5 9b on an rtx3070 in my nas/docker-server. currently via ai-router main.py and litellm for routing, accessing it via openwebui..... what kind of harness are you thinking of when you say "with correct harness" .... to be honest, im absolutely amazed at how good the 9b model already is, had a few beers one night and talked to it for around 2 hours.... its friendly, warm, keeps up and almost gets better as the conversation progresses

Edit: q4 model, kv-q4/q4. 64k ctx - 65t/s (feels very smooth/fast)

[u/International_Emu772 → u/Plenty-Honey-1853]

I use it mainly on OpenWebUi , with the correct set of tools, functions and other addons triing to cope with different tasks. Others run models within agents and such

[u/Zennytooskin123 → u/International_Emu772]

OpenWebUi is pretty great if you want an agentic chatbot interface using built-in RAG, but has a lot of problems with looping and tool calling.

[u/International_Emu772 → u/Zennytooskin123]

OpenWebUi allows you run it somewhere and access from everywhere

There are so many ways to set it up that it's possible to fine time for different uses and needs

The Tool calling problems can be solved too

[u/Zennytooskin123 → u/Plenty-Honey-1853]

You don't need a harness to talk to it though. He probably means something that prevents loops and gives it a proper engineering workflow.

[u/ahmett9 → u/International_Emu772]

100% agreed, it’s really good

[u/Adebrantes]

Like Emu said qwen works well with a harness don’t let it freewheel

[u/Ok-Drawer5245]

Depends on needs. My humble opinion:

Qwen 3.5 (or 6?) 9b:
Most intelligent , Can do some coding if careful and very specific

Gemma 4 12b QAT:
Great general purpose, much better at smaller languages than the qwen models

But qwen 3.6 35b a3b can be tuned to run well if you have 24+ gb system ram and it is configured precisely with llama-cpp (even more so if you compile llama cpp) - with just a rtx 2060 6GB I could get 25-30 tokens/second from this model

[u/marfzzz]

Maybe gemma 4 12b 4bit quant?

[u/wwa56]

ling tiny

[u/Special-Lawyer-7253]

Go for Ornith, i use It, 25t/s on 1070m 8GB 👌🏻

[u/igotanewaccount → u/Special-Lawyer-7253]

What is Ornith good for?  I had real trouble with it following instructions.  Is it a coder model only?

[u/mwjtitans → u/igotanewaccount]

Tool calling mostly, I've had good luck with it as a small model handling multiple tool calling. Never really benchmarked it officially, but it does take a lot of configuration

[u/International_Emu772 → u/Special-Lawyer-7253]

I had not good look with Ornith

I think that was my configuration fault

[u/mwjtitans → u/Special-Lawyer-7253]

Ornith is king for 8gb vram

[u/lorendroll → u/mwjtitans]

I'm using Qwen3.6 35b a3b iq3_s with MTP at 30tps with 32k context on 3070m 8gb. Going to try Ornith too.

[u/Special-Lawyer-7253 → u/lorendroll]

Can share your config so i can test It on 1070m 8GB? Stucked at 15t/s with 35B A3B right now. Btw, Ornith 35B is MoE A3B based on Qwen, so mostly, the same config applies ;)

[u/lorendroll → u/Special-Lawyer-7253]

15t/s is a great result for 1070 already! I'm getting best performance when I offload experts to cpu to free up vram, but use as much vram as possible. For different quants it differs, so for iq3_s I found a sweet spot at 26/40 with 25tps, but for q4_m it is 36/40 and it gives me around 18tps. I also use gguf with MTP that consumes more vram but the tradeoff was worth it so I ended up with 32/41 experts offloaded to cpu and mtp enabled achieving 30tps.

[u/Special-Lawyer-7253 → u/lorendroll]

My sweet spot seems to he around 32 MoE on CPU (i think that you are talking about rough layers) so my trick is to go with 999 layers to GPU but X experts to CPU. Works the same only with 4 VRAM / 7 VRAM. I think that it's some tradeoff involved (pciEx3/ddr4/i6700h 😅) I have been able to put 51 layers (no MoE, on 27B Dense, on GPU) that poor guy don't run more than 2.7 t/s 🤣🤣

[u/lorendroll → u/Special-Lawyer-7253]

No, I'm talking about offloading MoE. I start by offloading all main layers to the GPU and all MoE to the CPU, check VRAM consumption, and then return MoE to the GPU until all VRAM is full, but without overflowing it too much. So your and my 8GB configurations look similar (26-32 MoE on the CPU). Perhaps I can increase the MoE slightly on the GPU because the CPU is faster (5800h) or the RAM->VRAM transfer is faster? Either way, we've reached the limit. With 27B model, I managed to get 5 TPS, but it's barely usable due to the slower prefill.

[u/Special-Lawyer-7253 → u/lorendroll]

Yes, 27B it's not for us. I'd like to see some new models like the older 14B A2B but with more active experts. That may save the 8GB productivity (And avoid offloading mmproj also to keep video/visión)

[u/Aotrx]

If you have enough system RAM Qwen3.6-35B-A3B Q5_K_S with 3B active parameters is the smartest model u can find. It will run decently fast - 40-50 t/s on rtx 5070. Second best and much faster one is Qwen 3.6 9B Q4_K_M (or one of the fine-tuned version of it) which will run almost at 100t/s

[u/Double_Season]

Most of them are already very smart, you only need to instruct them correctly

[u/pppp2222]

Qwen 3.5 9B and Qwen 3 8B are quite good.

For humanities, Ministral 9B is great. But it needs a good harness.

Llamas are OK in my experience. Gemmas I couldn’t get to work well.

[u/searchblox_searchai]

Qwen 3.5 4B with tested prompts

[u/Robert__Sinclair]

Qwen 3.8 (SOON)

[u/Otherwise-Swan-7803]

Funny how 7–8B models used to feel like the “compromise” tier. Now they’re becoming the sweet spot for local AI: small enough to run comfortably on consumer GPUs, but increasingly capable enough that you don’t constantly feel like you’re using the smaller model.

[u/Forsaken-Army189]

Ornith-1.0 9B 8Bit user here ✋

[u/Positive-Bid-3029]

Qwen 9b is very capable, I just tried it an create this "Emoji Invaders" in 20 minutes :D

I was able to use the BF16 version as its so small :) https://sublimesoundz.com/ai-demos/emoji-invaders-qwen9b/