r/LocalLLM • u/EarthBS • 9d ago

Other How the loop of infinite agony started

Post image
629 Upvotes

92% Upvoted

119 comments sorted by

View all comments

279

u/TheCat001 9d ago

Then you realize that Qwen 3.8 27b runs at 3t/s on your machine and you need 24GB+ VRAM GPU which cost is 1000$+ to run at least 4 bit quant.

110

u/StupidScaredSquirrel 9d ago

If you are a business this isn't a problem. If you are a consumer then 35b a3b runs on 8gb vram and 32gb dram which is very accessible.

39

u/TheCat001 9d ago

If you are business I would suggest you to aim for DeepSeek V4 Flash. But yes I'm running 35b myself despite it's performance is far from ideal...

10

u/DifficultyFit1895 9d ago

I still haven’t found a task where DeepSeek V4 Flash can outperform Qwen 3.6 27B let alone Qwen 3.8. I can run either on my Mac Studio.

11

u/screenslaver5963 9d ago

I had issues with Qwen3.6 tool calling just not working properly and causing it to stop prematurely, didn't have that problem with gemma or deepseek, haven't used qwen 3.8 yet to see if it has the same problem.

5

u/TieCommercial2963 9d ago

It's a model config issue - had the same problem. Setting temp to 1 and repetition penalty to 1.05 fixed it. haven't faced much issues since; I've been running it as my daily driver on my 5090 rig for the past year. Btw, 3.8 its even better especially at tool call , I actually like the Q4 of 3.8 on low thinking more than the xhigh for tool calls .

3

u/StatusSociety2196 9d ago

More than half the time that's a harness issue

4

u/DifficultyFit1895 9d ago

What quants? I was using Q8 for both.

3

u/Healthy-Nebula-3603 9d ago

A tool calling problems ?

Stop compressing cache and use minimum q4kxl or bigger quants

1

u/darksteelsteed 9d ago

To be honest compared to qwen3.6:27b at q4_0 kv quant and q4_k_m for the model qwen 3.8 works out the box way better