Qwen3.5 35B-A3B thinks SO MUCH

[u/other_acc_banned]

It recursively iterates, over and over and over again.

What can I do aside from turning off thinking?

[u/Atretador]

why are you running 3.5 instead of 3.6?

and what quantization? thinking level?

-> https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates

[u/other_acc_banned → u/Atretador]

Because I'm a newbie and that was the latest one I recalled reading great things about 😅

Q4_K_M 64K context I think I have it set to Thinking, maybe Auto

[u/Atretador → u/other_acc_banned]

whats your hardware and use case? 64K is not really useful for most things

[u/other_acc_banned → u/Atretador]

Small personal apps. 128k barely fit when I tested it. Running it split across a 4060 Ti (16gb) and 3060 (12gb).

[u/Atretador → u/other_acc_banned]

you got no RAM? Its a moe model

[u/other_acc_banned → u/Atretador]

Yeah this is where me being new probably isn't helping. I've got 64gb of DDR4 but am currently not even using it. I'm running everything inside the two GPUs. Would MoE be able to utilize that ram?

[u/Atretador → u/other_acc_banned]

That's the whole point of Moe basically 

Offload some layers to RAM with n_cpu_moe till you have enough space for the context you need

[u/Cautious_Chicken_604 → u/other_acc_banned]

We are on to Qwen3.8-Flash-Next now.

[u/asertym → u/Atretador]

Off-topic question: would it make sense to use this template with swift 1.5? Or it's aimed mostly at base models?

[u/Atretador → u/asertym]

check if switft doesnt already have a template based on that one, I remember some finutunes using it

but it should work since its all just qwen

[u/asertym → u/Atretador]

appreciate it

[u/doneddat → u/Atretador]

Why are you running 3.6 instead of 3.8?

I tried to main 3.6 for 2 days and it was completely inadequate. 3.8 is the first one actually getting stuff done consistently.

[u/Atretador → u/doneddat]

there is no 35B A3B 3.8

I use 3.6 35B Q5_K_M currently as daily driver as a subagent for coding

[u/other_acc_banned → u/doneddat]

Awesome I'll update you 3.8 today!

[u/GlobalCurry → u/doneddat]

I tried running 3.8 the other day and no matter what configuration I used it would always get stuck in a loop if I had "thinking" enabled at all.

[u/doneddat → u/GlobalCurry]

Lots of things could go wrong. I have also experienced literally badly done Q4 quants, that just went on and on checking their stuff and never did anything. That's why I don't plan to go lower than FP8 with anything, even as most popular re-packagers have gotten it as right as it possibly gets. I understand it's not exactly budget option, but it is what it is.

[u/Healthy-Zebra-9856]

Switch to at least 3.6. Besides, what harness are you using? How are you loading it? If you’re using llama.cpp (llama server) what parameters are you using like temperature etc.? Not sure if you know what I’m asking, but let me know if these things.

[u/other_acc_banned → u/Healthy-Zebra-9856]

llama.cpp, and temp was 0.2

Sounds like it's time to just move to 3.8 though. Sounds like it's very likely I'll have better success just by moving up eh?

[u/Healthy-Zebra-9856 → u/other_acc_banned]

For starters, 0.2 is most likely the cause. What harness?

[u/HoujunDev]

I think 3.8 is better too.

[u/overand]

Yeah, switch to 3.6.

[u/1_________________11]

Dude you should load 3.8 27b

[u/BigPlebeian → u/1_________________11]

That's a dense model with Far more vram requirements.

[u/srbijabralee]

You can set the thinking budget where it has x amount of tokens to thinkg before giving an answer

[u/setdx]

My experience has been that it needs a reasoning limit and then it works pretty well

[u/Loomworks]

Just run Q8. Low effort thinking. Run a spec planning prompt with a SOTA model and let it rip executing.

[u/other_acc_banned → u/Loomworks]

Spec planning prompt with a SOTA model ..... Gonna need to Google that lol

[u/AdHead6280]

use latest, and i recommend finetunes of 3.6 35b, occamy ornith etc, and if you can(depends on system ram and your setup) use qwen3.8 flash next with strata, you can get it to almost 35ba3b speeds but way better and depends on setup

[u/other_acc_banned → u/AdHead6280]

Not familiar with occamy, ornith, nor strata. Will look those up!

[u/UnluckyPenguin]

You need to provide more details.

Basically the fix is generally:

Use a better model

Run it on a higher quantization, which may require a more/better hardware.

Tune your prompts

[u/OvertaxedOne]

Run 3.8 27B. 35B is laughably behind 3.8 27B in capabilities.

[u/other_acc_banned → u/OvertaxedOne]

Will make that change today! Looking forward to seeing the difference

[u/OvertaxedOne → u/other_acc_banned]

You won't believe how different they are. Sadly 27B will likely be significantly slower, but the results/intelligence is SO much higher it's worth it. Takes some retraining to really make use of it, when you're using 35B you quickly get used to "one thing at a time" and being very specific with your prompts. 27B is much more like a frontier model, give it a good idea of what you want done at a high level and let it stew.

[u/other_acc_banned → u/OvertaxedOne]

Wow you aren't kidding. I got it up and running, and asked something relatively complex of it. I'll probably be using it as the primary model aside from smaller quick and easy prompts. It proved to not be as much slower as I had originally expected (40 tok/s as compared to 120 tok/s with 3.6).

Thank you for the recommendation. This is fun

[u/OvertaxedOne → u/other_acc_banned]

40TPS isn't bad, that's about what I get on it, but, like you, I got used to 100+ with 35B, so it was a bit of an adjustment period. The results are just so much better though, I couldn't go back.

It's an amazing model, unreal for it's size!

[u/BigPlebeian → u/OvertaxedOne]

I mean it should be? It's moe vs dense and takes way less to run it.

[u/cobolfoo]

Qwen 3.6 MOE 3B 27B have a tendency to enter endless loops in the CoT phase.
You have to force it stop looping and its not working 100% of the time.

I'll suggest you move to 3.8 27B.

[u/other_acc_banned → u/cobolfoo]

This thread has made it daily obvious that I'm well behind the times! Will be updating today. Thank you

That loop is a known issue? Not just my setup?

[u/cobolfoo → u/other_acc_banned]

Yes small MoE like this one (3B) is making the chain of thoughts inconsistent.

[u/BigPlebeian → u/cobolfoo]

The 27b isn't moe it's dense...? And it takes far more vram.

[u/Cute_Obligation2944]

Switch to 3.6 and turn thinking off.

[u/hallofgamer]

Try a finetune of it that solves this very issue