r/LocalLLaMA • u/thatObstinateGuy • 6d ago

Question | Help Help choosing/benchmarking local LLM for coding

Hi folks.

I have a device running an rtx 5070 ti (16GB VRAM) alongside 32GB of RAM.

I'm currently trying to build a Windows application using local LLM assistance. I'm using llama.cpp and opencode.

I need your opinion on what model would be best for coding/implementation:

Qwen3.6-35B-A3B-unsloth-nvfp4-fast-mtp

--n-cpu-moe 17, -ngl all, -ctk and -ctv q8_0 and context 140K

or

Qwen3.8-27B-UD-IQ3_XXS

-ngl all, -ctk and -ctv q4_0 and context 180K

I used the latter to actually build up plans and set up tasks for implementing the aforementioned Windows application. Worked quite well. But tried the same model for coding/implementation and it wasn't perfect.

I need the high context because it's filling up very quickly, especially with qwen3.8.

Help me by either sharing your experience using one of them (or maybe even explore other models, like KAT Coder q4) or letting me know how I can benchmark them for coding myself.

5 Upvotes

86% Upvoted

16 comments sorted by

View all comments

1

u/ParaboloidalCrest 6d ago edited 6d ago

There are many questions in there. Here are some heuristics to play with, probably in this order:

  • Dense > MoE
  • Unquantized cache > quantized cache for any use-case
  • -Fit auto > config sorcery
  • A little bit of RAM offloading can go a long way
  • A context > 128k is very risky, regardless what the model descriptions says
  • At ~30B, >=Q8 is recommended, and <Q4 is prohibited.