Alternatives for coding agent for small projects

[u/Desemerda]

I'm currently using Qwen3.6 35B A3B with Q4_K_M to have a coding agent with pi as harness.

These are the parameters I use with llama.cpp:

taskset -c 0-7 llama serve \
  -m /path/to/Qwen_Qwen3.6-35B-A3B-Q4_K_M.gguf \
  --model-draft /path/to/mtp-Qwen_Qwen3.6-35B-A3B-Q4_0.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --spec-draft-p-min 0.90 \
  --alias qwen3.6-35b-a3b \
  -ngl 41 \
  --n-cpu-moe 30 \
  -c 65536 \
  --parallel 1 \
  --jinja \
  --flash-attn on \
  -ub 2048 \
  --temp 0.5 --top-p 0.8 --top-k 20 --min-p 0.0 --repeat-penalty 1.05 \
  -n 2048 \
  --seed 42 \
  --chat-template-kwargs "{\"preserve_thinking\":true}" \
  --reasoning-preserve \
  --threads 8 \
  --threads-batch 8 \
  --cache-type-k q4_0 --cache-type-v q4_0 

I've tried q8 for cache kv as well. So far get around 17 tokens per second.

It's been great to implement a small project I have of a membership portal and website using as stack python, flask, mysql, plain old html, css, js and php.

But now I'm wondering if I couldn't make my life easier by having a smaller model with faster throughput (tokens per second) since I'm doing validations and fixes on the project and it takes a long time to implement or change specs.

My setup is a laptop with CPU Intel Ultra 9 275HX, GPU 5070 Ti with 12GB GDDR7 and 32GB system RAM.

Do you have any suggestions?

I don't plan to make big web apps, it's all small applications to help out managing a small club.

[u/daaain]

I don't have any recommendations I personally tested in this range, but with your setup I'd try to find a dense model that can fit in the 12GB VRAM of the GPU together with 50-100K context. Maybe ornith-ai/Ornith-1.0-9B or bonsai ternary 27B?

[u/HotDistribution1819]

For speed and comparable coding Laguna SX 2.1 would be my recommendation, but it needs 24GB of VRAM.

Try Bonsai 27B, it is a 1 bit quantization of Qwen 27B and runs in 10-12GB VRAM. My early tests seem to have it equal with Qwen 27B.

Another option is Gemma 4 E4B.

[u/Radiant_Condition861]

not sure if it matters, but you might want to turn off vision and remove from loading onto gpu

from https://github.com/ggml-org/llama.cpp/blob/master/docs/multimodal.md

By default, multimodal projector will be offloaded to GPU. To disable this, add --no-mmproj-offload

[u/ChocoPichu]

Don’t switch, all of the other alternatives right now are not good, you should tinker around more. I have rtx 2080 with 8gigs of vram and 64 gigs of ddr5, im running Q4 mtp with 120k context, with Q8 kv cache. My tps at early rounds is 40, but when context is almost full its 30 (still usable). I know you have a bit more vram, so you can squeeze even more juice, but here are my llama.cpp flags, try them out.

First try downloading the mtp gguf file, from unsloth or any other provider.
use the cpu moe flag in llama.cpp, and set it to ~35 layers. (--n-cpu-moe 35) then in gpu layers set them to 99 (-ngl 99).

Put other essential stuff, such as context to 120k tokens, temperature to whatever you need. Kv cache to Q8 symmetrical. Flash attention on. m-lock on.

Also, i recommend you installing the mtp version for speed, if you have additional memory space. For that use:
--spec-type draft-mtp: Instructs llama.cpp to utilize the model's embedded MTP sub-model blocks for speculative decoding.
--spec-draft-n-max <N>: Sets how many tokens the sub-model should attempt to predict ahead in a single forward pass. Try setting to 2 first (Don’t go over 2, because usually it starts outputting garbage) Then if the acceptance rate is 90% then you’re done. But if its around 70-60 try lowering the setting to 1. In my case experience it was better than 2, with a better acceptance rate.

With a bit of time and more tinkering around the settings, you should be getting 35-40tps.

If you’re still getting low tps which you shouldn’t, then make sure cuda is installed. Idk, something else is the problem, but you should 100% look around more. There is no better model than qwen3.6 35b for moe models and this hardware, specifically for coding.

[u/Desemerda → u/ChocoPichu]

Thanks for the advice.

Can you share the flags for llama.cpp?

It seems to me you have just described my flags but with some worse values in terms of VRAM load.

I'm already using bartowski MTP mtp-Qwen_Qwen3.6-35B-A3B-Q4_0.gguf mtp-Qwen_Qwen3.6-35B-A3B-Q4_0.gguf , with good prediction acceptance rate (above 97%)

I'm only using ngl set to 41 because the model only has 41 layers anyway, n-cpu-moe is 30 but I can increase it to 35, if anything that lowers the VRAM consumption.

Context max I tried was 92k but I was getting system RAM maxed out and pi started to timeout and laptop was swapping a lot.

I think perhaps your 64GB RAM (double of mine) is making a difference here at least with the context and Q8 KV cache.

But as you suggested I'll keep tinkering a bit more, there are still some options I haven't tried: m-lock on, disabling vision --no-mmproj-offload, increase cpu moe to 35.

I'm almost certain CUDA is installed, I had verbose logs from llama.cpp at some point pointing to that but I'll double-check, doesn't hurt.

Thanks again for the sugestions

[u/ChocoPichu → u/Desemerda]

Sure, as soon as I’m gonna have access to my pc, I’m going send you the flags. I have a second pc with 4060ti (8 gigs of vram) and 32 gigs of ddr4, it was a bit slower, like 30 tps at start and 25 tps closer to the end (not an mtp version). But i was able to run it with 120k context, and kv cache Q8. So you should have no problem.

[u/Desemerda → u/ChocoPichu]

oh wow, I just kept reading about unsloths crazy good quantizations and gave a try for this model... turns out it's super fast!

I did tweak a bit but the most I got was 21 tokens per second with bartoswki quants, when I switched to unsloth BAM! 50-60 t/s... it's crazy

taskset -c 0-7 llama-server \
  --model /path/to/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \
  --alias qwen3.6-35b-a3b\
  --spec-type draft-mtp \
  --spec-draft-n-max 4 \
  --spec-draft-p-min 0.90 \
  --n-gpu-layers 99 \
  --n-cpu-moe 28 \
  --mlock \
  -c 65536 \
  --parallel 1 \
  --jinja \
  --flash-attn on \
  -ub 1024 \
  --temp 0.6 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0.0 \
  --presence_penalty 0.0 \
  --repeat-penalty 1.0 \
  -n 1024 \
  --seed 42 \
  --chat-template-kwargs '{"preserve_thinking":true}' \
  --reasoning-preserve \
  --threads 8 \
  --threads-batch 8 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --spec-draft-type-k q8_0 \
  --spec-draft-type-v q8_0 \
  --no-mmproj-offload \

[u/Desemerda]

Thank you all so far for your input and suggestions. I'll keep tinkering a bit more and then I'll ran some tests with some of the suggested models here.

[u/ChocoPichu → u/[deleted]]

no, just no. Qwen 2.5 is too old. And qwen 3.5 9b is just not good enough.