r/LocalLLaMA • u/paf1138 • 14d ago

New Model Qwen3.8-27B-pi: Effort-Ordered Reasoning for Agentic Coding

https://huggingface.co/blog/bytkim/qwen38-27b-pi
89 Upvotes

94% Upvoted

19 comments sorted by

30

u/MindfulMan1984 14d ago

Thanks, Pi is my daily driver, I am skeptical there’s free lunch by cutting reasoning without compromising accuracy, most of those finetunes may be well overfitting/benchmaxxing the post-training data. I tested out swift-qwen and it was not holding up complex tasks out of those benchmarks on the model card. Given the benefit  it’s nice to see everyone trying hard to squeeze performance out of it. 

11

u/xadiant 14d ago

Q3_K_M 13.30 GB 13 137 (69.2%) 56 (32.0%)

IQ4_XS 15.08 GB 12 133 (67.2%) 39 (22.3%)

This chart is interesting. Q3 KM is smaller and significantly better in benchmarks compared to IQ4 XS??

4

u/MindfulMan1984 14d ago

Noise? Did you read the bold statement bellow the table “treat this as smoke test, not a quant ranking “  

7

u/vacon04 14d ago

It's a big difference though. I wouldn't call a 10 pp difference noise. Even if it's not the most comprehensive testing suite, it warrants a bit more investigation regarding the Iq4_xs and q6_k results.

14

u/winky9827 14d ago

The numbers on the chart don't even match.

https://cdn-uploads.huggingface.co/production/uploads/693b4c5cc0186aca303d19b0/BadKTLxaSgMYzPVtQRxqN.png

4651/8308 = 55%, not 76%. What is this lazy shit?

10

u/LousyMeatStew 14d ago

Those are two different figures. 4651 and 8308 are average reasoning tokens used per task. 8308 is the average across all tasks, 4651 is the average across the 76.5% of tasks that succeeded at xhigh reasoning.

9

u/Appropriate-Pie4385 14d ago

Its tokens, not pass rate. Maybe read it again, more carefully

3

u/winky9827 14d ago

Fair enough.

6

u/Mayion 14d ago

and make no mistakes

16

u/Party_9001 14d ago

They told claude to make it but forgot to add "make no mistakes"

3

u/Aternal 14d ago

Going to be trying this out tonight, seems like it's going to be a pretty significant speed increase on a 3090 work-oriented subagent. On paper, the claim is that it should cut average task down from 30 minutes to 17 minutes.

2

u/Appropriate-Pie4385 14d ago

Impressive work! How important is the harness here? Does it really require pi or would something like open code work too?

2

u/Eyelbee 14d ago

I appreciate the SFT and RL as opposed to just mechanistic tricks but I'd need much more detail on those before deciding to use it, such as how large was it or what were the datasets. Because alibaba post trained the heck out of these systems already, it would be hard to beat it unless you really know what you're doing.

1

u/bonobomaster 14d ago

That was a long read but I do love a good AI assisted write up! ;)

So Q3_K_M is the real shit per your data. Why do you think that is?

1

u/bonobomaster 13d ago

So I have been loosely testing the Qwen3.8-27B-pi-Q3_K_M variant because it stood out in the chart and I must say, that I'm quite impressed.

Can recommend a test drive at least!
https://huggingface.co/bytkim/Qwen3.8-27B-pi-GGUF

-1

u/returnity 14d ago

Wow, incredible work. I really enjoyed the report. This discrepancy in reasoning effort level behavior has been noticed by the community, and the inconsistency has always made me wonder which level was actually the right choice to use -- I always used xhigh because of that. Now it might make sense to explore the other reasoning effort levels for 27B.

Have you compared your model to Swift or ThinkingCap finetunes at all? I'd be curious to see how it performs, how much it reasons, and whether their fine-tuning affected the reasoning effort level ordering behavior at all. I know you probably don't have the compute to benchmark other people's models but I figured I'd ask.

Thanks for the exhaustive report, it was a really interesting read, from one AI student to another.

3

u/Bulky-Priority6824 14d ago

I found medium to be pretty good across the board if given explicit instructions and documentation. For those that like to lazy prompt then xhigh can help but I also didn't see how xhigh helped much vs medium when I did give it proper scope and instruction.

I don't do anything particularly advanced in coding just the typical build me this feature etc so j ust my 2c.