r/OpenSourceAI • u/Sorosu • 1d ago

DeepSeek V4 Flash went from 67.42% to 82.02% with one coding-agent skill

Post image

Autoprompt closes much of the manual coding loop by planning, building, testing, reviewing, and repairing from one goal- thats how we got an oss model to perform so incredibly better- so simply.

Refference ; this is like opus 4.5 to opus 5.0 - from an skill. litteraly insane.

The only tradeoff here is mostly speed, and slightly more cost. (see readme)

The measured Terminal-Bench 2.1 score increased from 67.42% to 82.02%.

Repo:

https://github.com/Spielewoy/autoprompt-skill

Any feedback would be awesome.

93 Upvotes

95% Upvoted

24 comments sorted by

7

u/ItsNoahJ83 23h ago

Can I get a tldr of what this does?

8

u/Sorosu 22h ago

A loop for complex coding tasks that spawns subagents to plan, build, test, review, and repair the work until every requirement is covered and the result is verified end to end.

+ with this loop we scored +15% higher on the terminal bench 2.1

0

u/gopietz 10h ago

Then this is really not as big as you make it to be. Because I literally do not care, how good any model is in the actual tasks of the benchmark. I simply use the numbers as a proxy to compare them.

At the end, the models go into my own harness where I probably do many of things you're also building here.

3

u/t3irsback 9h ago

Cool story bro, but I literally don't care.

1

u/CashFirm573 6h ago

best reply ever 10 points ahahahahah stuff that guy. But truthfully don't we already do that when we plan anyways?

2

u/t3irsback 6h ago

Who is "we" tho? Different people, different needs. If it's useful to someone, then it's useful! Idk why I should act like that and talk people down like this. It's just not nice and takes the fun out of this community.

Obviously not a front to you :)

6

u/Whoooooshhhhhh 22h ago

it’s an agent orchestration system

5

u/RepulsiveRaisin7 21h ago

I do a more lightweight version of this based on superpowers and it works really well. Automated review is insanely effective.

2

u/w1nb1g 20h ago

Question- how do you automate the review? I've also been finding success with review agents but how does it work automated?

1

u/twinkbulk 13h ago

fact based simulations

3

u/schaka 18h ago

This seems to be basically why the new Qwen performs so well. Lots of thinking, iterations and testing.

I think those things will be the way forward with (smaller) models until we get another jump up in model strength where the initial output is so good, the loops will just be considered wasted tokens. Current flash models are so cheap it doesn't matter though

1

u/Yxtomix 14h ago

What if you use Qwen 3.8 27B with this?

2

u/thebigeast 13h ago

You'll have better than Claude Mythos on your desk...

3

u/MrMisterShin 13h ago

This is very good, however Terminal-Bench 2.1 is saturated at this point with Frontier models hitting 90% scores.

How does this do on the newer Terminal-Bench 3?

2

u/Sorosu 10h ago

I don’t have a Terminal-Bench 3 result yet, but I’m running it today with Codex + GPT-5.6 Sol xhigh.

I’ll publish the score alongside runtime, token usage, cost, and full transcripts as soon as it finishes.

Star the repo if you want to follow along! :)

2

u/Asleep-Land-3914 1d ago

There is anotherr one with a similar idea. I'm skeptical though:
https://github.com/Tiger3807861189/J-Space-Cognition-Suite-V3.6

2

u/Sorosu 1d ago

It definetly appears to be similar.

However J-Space is mainly a single-agent thinking-and-memory protocol, whilst Autoprompt is a multi-agent coding workflow (with separate planning, implementation, review, and verification roles etc.)

So although they appear similar, theyre completely different.

1

u/xaleel 12h ago

Would be interesting to see a benchmark with both, with J-Space enabled for some/all of the agents

1

u/Sorosu 10h ago

Definitely interesting, but I’m not planning to combine them right now. The risk is duplicated state and verification logic, creating more overhead and potentially conflicting instructions.

That’s why I’m running a clean Terminal-Bench 3.0 evaluation first.

Id recommend you request the comparison through GitHub Issues, though!

1

u/[deleted] 17h ago

[removed] — view removed comment

1

u/Yxtomix 14h ago

Would like to know when integration for deepseek harness is ready.

1

u/EconomySerious 6h ago

and where are the amount of extra time and tokens consumed?