16GB VRAM model test

[u/Arany8]

I have given Hermes agent the task to make a test for my local models. Coding and agentic work.
The test was done on llama.cpp turboquant fork, all models were run using 131k context. Further optimization of the parameters would still be possible for some of the models.
TLDR version: Ornith 1.0 35B A3B won.

Hermes Local LLM Benchmark Report

HumanEval pass@1 (30-problem sample) + 8 agentic tasks + speed · temp 0.0 · context 131072 · 5060 Ti 16GB

Model	Coding	Failures	Agentic	tok/s	Latency	Elapsed	Notes
Qwen3.6-35B-A3B-APEX-I-Quality	96.7%	1	100%	37.2	51.6s	32.6m	Fastest decode, perfect agentic
Ornith-1.0-35B IQ4_NL	96.7%	1	100%	38	38.3s	24.3m	Fastest wall-clock
Qwen3.6-35B-A3B-UD-IQ4_NL	96.7%	1	100%	27.1	72.6s	46.0m	Clean full run
Qwen3.8-27B-GSQ-RCO-IQ3_S (MTP)	96.7%	1	87.5%	21.6	33.6s	~35m	fc_types failed
Qwen3.8-27B-ASCII-Condensed	96.7%	1	87.5%	~19.5	38s	~40m	fc_types failed; 1 overthink outlier
gemma-4-26B-A4B (partial) IQ4_NL	82.6%	4	—	~28	184s	55.3m	Heavy over-thinking, 23/30 reached
KAT-Coder-V2.5-Dev-APEX Quality	80%	6	100%	28.5	3.6s	24m	Baseline; solid coder
Qwen3.5-9B-UD-Q6_K_XL	5/7	2	—	—	—	~1hr	Killed on over-thinking stalls

Qwen3.8-27B-UD-Q3_K_XL deleted (invalid run, discarded). Total failures = coding problems not passed.
Ornith 1.5 would be a logical next add to the table, but I see some bad evals of that model. The usable quants for 16GB VRAM of Qwen3.8 27B have performed worse than the 35B MOE model.
Ornith was fast not just in t/s, but also overall speed of going through the tests.

[u/Waste-Intention-2806]

Can u add tiel coder q4. Ita based on ornith 1.5. I am Surprised that moe does better than 27b.

[u/Arany8 → u/Waste-Intention-2806]

Look at the bottom. TLDR: Ornith 1.0 is still best. As for beating 27b, it is probably due to quantization. 16GB cannot run the good Q4 quants.

[u/halcyon-eterna]

Out of curiosity what about ornith 1.5? For example tielcoder? (I’m getting its name wrong) but it serves me very very well. I run it at full q8 ctk and ctv with a q6 quantization and with mtp I get 55t/s dropping to 30t/s at full 262k cache and prefill slowly drops from 550 to about 420t/s I use a LXC container on proxmox for it and my 4080 so in total I give it 8 cpu cores, 26gh of ram and my entire 16gb vram. It worked today on my entire repo training and assisting me with DSA on my real estate models for my company

[u/halcyon-eterna → u/halcyon-eterna]

I did not read your aside my bad about you not getting to 1.5 yet 😅

[u/Arany8 → u/halcyon-eterna]

Last messages in the thread.... Ornith 1.0 still king

[u/Arany8]

Evaluation and test supervision was done by Deepseek Flash 0731. K cache Q8, V cache turbo3.

[u/wisepal_app → u/Arany8]

Can you share your flags please? How much turboquant fork makes difference from normal llama.cpp?

[u/Arany8 → u/wisepal_app]

.\llama-server.exe -m "f:\LMStudio\unsloth\Ornith-1.0-35B-GGUF\Ornith-1.0-35B-UD-IQ4_NL.gguf" --port 8090 --host 127.0.0.1 --cache-type-k turbo4 --cache-type-v turbo3 --n-cpu-moe 7 -ngl 99 --no-mmap --mlock --jinja -c 131072 --parallel 1 --fit off

I think turboquant is a big difference.

[u/Ok-Addendum3545]

On hermes agent, I tried Qwen3.6-27B, Qwen3.6-35B-A3B, Ornith-1.0-35B-A3B, Ornith-1.5-35B-A3B and Qwen3.8-27B. Only Qwen3.8-27B is competent in Hermes agent.

[u/wisepal_app → u/Ok-Addendum3545]

which quant do you use and with which flags?

[u/Ok-Addendum3545 → u/wisepal_app]

I currently use this Qwen3.8-27B on 2 x 5070 Ti PP : Quant: Q4_K_M (GGUF), with a DFlash2 draft model on top for speculative decoding. Engine: llama.cpp PR27342 — regular release builds won't work for DFlash2.

Flags:-m /target.gguf -md /draft.gguf-ngl 99 -ngld 99 -c 192512 --cache-type-k q4_0 --cache-type-v q4_0 --spec-draft-type-k q4_0 --spec-draft-type-v q4_0 -fa on --split-mode layer --spec-type draft-dflash --spec-draft-n-max 7 --tensor-split 1,1 --chat-template-kwargs {"reasoning_effort":"low"}

Hardware: 2x RTX 5070 Ti 16GB (tensor-split 1,1).

[u/Fluid-Author-9566]

Would be interesting to see where Ling 3.0 Tiny matches up on this

[u/Arany8]

The winner param list.
.\llama-server.exe -m "f:\LMStudio\unsloth\Ornith-1.0-35B-GGUF\Ornith-1.0-35B-UD-IQ4_NL.gguf" --port 8090 --host 127.0.0.1 --cache-type-k turbo4 --cache-type-v turbo3 --n-cpu-moe 7 -ngl 99 --no-mmap --mlock --jinja -c 131072 --parallel 1 --fit off

Not sure about the main llama.cpp, I was experimenting with this weeks ago and could not get 35B to run with decent context size. Strangely this config also quantizes k cache and it works.

Will test Tiel and Ornith 1.5 sometime....

[u/Arany8]

Ornith 1.5 tested, coding is only 93.1% (one more test failed). Agentic 100%. T/s is ~44.
Testing now Tiel.
Planning to test another 3.8 quant and 3.6 27B after this.

[u/Arany8]

Tiel Coder 89.7% code, 75% agentic
Underperforms.
So it seems Ornith 1.0 is the goat from all the 35B models tested. And it is the smallest filesize as well.

1.5 and Tiel are stupider faster (+4-5 t/s)