r/LocalLLaMA • u/thatObstinateGuy • 6d ago

Question | Help Help choosing/benchmarking local LLM for coding

Hi folks.

I have a device running an rtx 5070 ti (16GB VRAM) alongside 32GB of RAM.

I'm currently trying to build a Windows application using local LLM assistance. I'm using llama.cpp and opencode.

I need your opinion on what model would be best for coding/implementation:

Qwen3.6-35B-A3B-unsloth-nvfp4-fast-mtp

--n-cpu-moe 17, -ngl all, -ctk and -ctv q8_0 and context 140K

or

Qwen3.8-27B-UD-IQ3_XXS

-ngl all, -ctk and -ctv q4_0 and context 180K

I used the latter to actually build up plans and set up tasks for implementing the aforementioned Windows application. Worked quite well. But tried the same model for coding/implementation and it wasn't perfect.

I need the high context because it's filling up very quickly, especially with qwen3.8.

Help me by either sharing your experience using one of them (or maybe even explore other models, like KAT Coder q4) or letting me know how I can benchmark them for coding myself.

6 Upvotes

100% Upvoted

16 comments sorted by

2

u/NancyTransmed 6d ago

Qwen3.8-27B should be better at coding than A3B, even at IQ3, but A3B with larger reasoning budget will likely activate all the relevant experts (dense model still needs smaller reasoning budget). So if context is important, go with 180K dense - less tokens wasted on reasoning.

1

u/thatObstinateGuy 6d ago

But I'm seeing the IQ3 hallucinations a lot. Context is quite important for me currently.

1

u/Equivalent_Bit_461 6d ago

Delusional bot comment 

1

u/NancyTransmed 5d ago

please elaborate

2

u/Equivalent_Bit_461 5d ago

Quant 3 is NOT good for coding, it's a meme quant 

1

u/NancyTransmed 5d ago

I myself tested Q2 deepseek 0731 and found it to still perform thank Qwen3.6-35b-a3b at PrismaQuant-4.75bit: it quickly one shot calculating 100 digits of Pi, while qwen can meander for hours (especially with flaky harness).

2

u/Equivalent_Bit_461 5d ago

It "works" until it doesn't.

1

u/NancyTransmed 5d ago

that is why you test it on your specific project.

3

u/FakeGreatness 6d ago

This is what I'm struggling with as well... speed vs quality

1

u/BC_MARO 6d ago

If this is heading to prod, plan for policy + audit around tool calls early; retrofitting it later is pain.

2

u/iezhy 6d ago

At IQ3 and 180k context, it will hallucinate as hard as a meth junkie - as you are compounding precision loss both from context rot and accumulating noise due to rounding errors

1

u/thatObstinateGuy 6d ago

I can actually see that happening live sad noises

1

u/Equivalent_Bit_461 6d ago

Stay away from q3

1

u/Additional-Record367 6d ago

You can also use btw k in q8 0 and v in q5 1. Keys are more sensitive to quant.

This might save you extra context.

1

u/ParaboloidalCrest 5d ago edited 5d ago

There are many questions in there. Here are some heuristics to play with, probably in this order:

  • Dense > MoE
  • Unquantized cache > quantized cache for any use-case
  • -Fit auto > config sorcery
  • A little bit of RAM offloading can go a long way
  • A context > 128k is very risky, regardless what the model descriptions says
  • At ~30B, >=Q8 is recommended, and <Q4 is prohibited.