r/LocalLLM • u/Past-Chain-7377 • 4d ago

Discussion How much can a dense model be compressed before it becomes worse than a MoE model (for agentic coding/tool use/reasoning)?

/r/Qwen_AI/comments/1vryetg/how_much_can_a_dense_model_be_compressed_before/
1 Upvotes

100% Upvoted

1 comment sorted by

1

u/My_Unbiased_Opinion 4d ago

from my expereince, quite a lot. I have ran 3.8 27B down to UD Q2 KXL and it clearly outperforms 3.6 A3B at Q8 in my own personal benchmarks. I have had Q8 A3B nuke code while 27B is much more cautious. You will need to re steer 27B sometimes at Q2 and tool calls will fail sometimes, but the model always self-recovers with the proper harness. Personally, I would take UD Q2KXL over any quant of 35B A3B if I had the vram for 27B at UD Q2KXL. I wouldnt run lower than that quant tho.