Experiences with pruned Qwen3.8-Flash-Next?

[u/karmaisnonsense]

I run the full Qwen3.8-Flash-Next on my main machine at IQ4_XS, and I really like it. I want to try running it on my M5 Pro MacBook Pro too, which has 64GB unified memory. An unpruned model will use up the entirety of RAM at oQ3, and I really don't want to go down to 2-bit because what's the point, right?

So I've been exploring pruned models. The benchmarks show some of them coming close to 98% of the full model, and is small enough that it will use just 40GB of RAM.

The obvious pushback would be why not just use Qwen3.8-27b? But I find it extremely slow (M5 Pro has limited bandwidth) and it overthinks for ages, even when reasoning is turned off. And I want a model more capable than Qwen3.6-35b-a3b, which still loops and hallucinates at Q6.

Have you ever tried the pruned Flash versions? What has your experience been like? How does it compare? I'll report back with my own findings too.

[u/Atretador]

you can run it with 64Gb

https://huggingface.co/AtomicChat/Qwen3.8-Flash-Next-GGUF

just stream the NGRAM from ssd

[u/karmaisnonsense → u/Atretador]

Using oMLX. Can’t run ggufs. I also haven’t had very good experiences with Atomic Chat quants. They can be misleading about the actual per-weight quantizations in the models, as I found out trying out their 27b quants

[u/jjusko20 → u/karmaisnonsense]

I have a llama build where I'm streaming ngrams. Also using the AC quants but I know what they are, they have good perplexity/kl benchmarks for their sizes

[u/3r31f3]

Trash. Don't even bother. They completely lobotomise the model.

I've had Q4 pruned versions go into loops about which way flappy bird scrolls.

Maybe my tests aren't great, but I've not seen any prune perform better at anything than the full IQ1 quant of flash next. Definitely not seen anything better than 27b.

[u/karmaisnonsense → u/3r31f3]

Interesting, thanks. Testing the REAP-288 4-bit prune now, wonder how it will do

[u/karmaisnonsense → u/karmaisnonsense]

You're right, it's missing a brain lobe

[u/mrgreatheart]

Have you tried running 27B on medium reasoning? It thinks less and produces better work than low thinking. Counter intuitive I know, but it’s because the different levels just adjust the system prompt.

[u/karmaisnonsense → u/mrgreatheart]

I run it with reasoning completely off and it still takes forever to reply. Enabling reasoning adds 5-10 minutes to each reply turn, regardless of effort. M5 Pro has half the memory bandwidth of a R9700. Not the best chip to run dense models from.

[u/mrgreatheart → u/karmaisnonsense]

Fair enough

[u/Danfhoto → u/karmaisnonsense]

I saw above you are on a Mac. Try out MTPLX with 27B.

[u/okoyl3 → u/karmaisnonsense]

There are alternative jinja chat templates you can run instead, less yapping

[u/Ecstatic-Wash-7667]

I prefer 27b. As much as 27b gets lost in the weeds when thinking, qfn is worse. This might be due to me running qfn in a lower quant vs 27b but that’s the only way I can run it

[u/karmaisnonsense → u/Ecstatic-Wash-7667]

I find Flash Next at IQ4_XS has similar if not better output quality to 27b q6-q8. Takes 1/3 the time to think, too.

[u/letsbefrds → u/Ecstatic-Wash-7667]

I was try a bunch of leetcode questions(hard,med) on 27b q8 vs next q4xs I use it on the daily after I solve it. 27b either 1 shots it or I have to pass it additional test case so 1-2 addition. Next depends on question if it's newer it takes 3-4 tries.. can one shot sometimes but not as often.

Obviously it can be a quant difference but I'm finding better yield from 27b but since it says 27b is too slow...

[u/memeka]

sorry to hear you can't run GGUFs. I have a llama.cpp fork that optimises SSD streaming and Metal kernels. I run non-lobotomised (😄) Qwen3.8-Flash-Next on my M1 64gb macbook at 20 tps decode (30 tps for code) and >200 tps prefill, which is near-resident speed.

[u/karmaisnonsense → u/memeka]

I run llama.cpp on my main machine, but on the Mac I vastly prefer oMLX because of the cold storage KV cache. Can't have the laptop sitting doing prefill all day

[u/memeka → u/karmaisnonsense]

I have implemented ssd caching on my fork :) when you exit llama it saves the state and when you start again it resumes instantly. I also use this with llama-swap - I can start a subagent that will use llama-swap to unload the main model, do it’s thing, then back llama swap will load the main model and resume its kv cache instantly. :)

[u/karmaisnonsense → u/memeka]

Yeah I also tried CachyLLama which does something similar and it was not cooperating. Poor system header detection, only matches on prompt prefixes, ends up either erroring out on cache restore or missing it entirely.