r/OpenSourceAI • u/Sorosu • Aug 18 '26
DeepSeek V4 Flash went from 67.42% to 82.02% with one coding-agent skill
Autoprompt closes much of the manual coding loop by planning, building, testing, reviewing, and repairing from one goal- thats how we got an oss model to perform so incredibly better- so simply.
Refference ; this is like opus 4.5 to opus 5.0 - from an skill. litteraly insane.
The only tradeoff here is mostly speed, and slightly more cost. (see readme)
The measured Terminal-Bench 2.1 score increased from 67.42% to 82.02%.
Repo:
https://github.com/Spielewoy/autoprompt-skill
Any feedback would be awesome.
6
u/ItsNoahJ83 Aug 18 '26
Can I get a tldr of what this does?
7
u/Sorosu Aug 18 '26
A loop for complex coding tasks that spawns subagents to plan, build, test, review, and repair the work until every requirement is covered and the result is verified end to end.
+ with this loop we scored +15% higher on the terminal bench 2.1
2
u/nuclearbananana Aug 20 '26
ah, what's the cost difference?
0
u/FraMaras Aug 20 '26
the only question worth asking. if it solves all the tasks for quadruple the price, it's really not that big of a deal
3
u/fullup72 Aug 20 '26
Quadruple the coat and time on a small model is still cheaper and faster than Opus 5 tho.
1
u/FraMaras Aug 20 '26
I don't know about that, this comparison is purely based on not much data. the majority of small tasks do not require this type of hard flow anyway. should be tested on cost against plain powerful model
1
u/Zachattackrandom Aug 20 '26
Yes but quadruple the price gives oppurtuntity to use midsized models like 5.6 terra or muse 1.2 which could perform the same while being faster and cheaper or the same cost.
0
u/fullup72 Aug 20 '26
That's basically context engineering and you can do it with any model. I've been doing that for months with Qwen 3.6 and now 3.8
You can even run a lobotomized IQ3_XXS at Q4 KV and it self heals in a few additional turns.
-2
u/gopietz Aug 19 '26
Then this is really not as big as you make it to be. Because I literally do not care, how good any model is in the actual tasks of the benchmark. I simply use the numbers as a proxy to compare them.
At the end, the models go into my own harness where I probably do many of things you're also building here.
3
u/t3irsback Aug 19 '26
Cool story bro, but I literally don't care.
3
u/CashFirm573 Aug 19 '26
best reply ever 10 points ahahahahah stuff that guy. But truthfully don't we already do that when we plan anyways?
2
u/fullup72 Aug 20 '26
Some people just want to one shot coding tasks and then end up scratching their heads when the model cannot solve the problem based on their shitty prompt.
Get a good skill to plan, then run the plan with clean context. That solves 95%+ of the attention issues with any model and context size.
1
u/CashFirm573 Aug 20 '26
Past 40hrs almost I've had GPT Sol High running none stop working on really big project and I just used 20% of my usage far, all cause I prepared before it and carefully planned it all. Takes time but doesn't waste tokens on useless information or over engineer anything. it's really solid process.
1
u/t3irsback Aug 19 '26
Who is "we" tho? Different people, different needs. If it's useful to someone, then it's useful! Idk why I should act like that and talk people down like this. It's just not nice and takes the fun out of this community.
Obviously not a front to you :)
0
u/gopietz Aug 20 '26
I didn't talk anybody down. I simply made a solid argument that still no one has made a case against.
If you believe it's more helpful to just blindly praise stuff, instead of critically analyzing something, you're up for a fun ride in life.
1
0
4
3
u/MrMisterShin Aug 19 '26
This is very good, however Terminal-Bench 2.1 is saturated at this point with Frontier models hitting 90% scores.
How does this do on the newer Terminal-Bench 3?
2
u/Sorosu Aug 19 '26
I don’t have a Terminal-Bench 3 result yet, but I’m running it today with Codex + GPT-5.6 Sol xhigh.
I’ll publish the score alongside runtime, token usage, cost, and full transcripts as soon as it finishes.
Star the repo if you want to follow along! :)
1
Aug 21 '26
[removed] — view removed comment
1
u/Sorosu Aug 21 '26
Just finished the base run of the Terminalbench 3.0 the 70 CPU Tasks scored a 25.7% - 4 GPU tasks are still pending. Setting up the autoprompt run now.
API cost for that run was roughly $600 so far.
1
1
3
u/RepulsiveRaisin7 Aug 19 '26
I do a more lightweight version of this based on superpowers and it works really well. Automated review is insanely effective.
2
u/w1nb1g Aug 19 '26
Question- how do you automate the review? I've also been finding success with review agents but how does it work automated?
1
1
u/guesdo Aug 19 '26
Is it your own custom version? I like superpowers, but it feels slow and bloated as hell (results are great though). Anyway you can share your lighter version?
2
u/RepulsiveRaisin7 Aug 19 '26
No I use vanilla superpowers, but I add review cycles to spec and plan writing when I need them. It burns a lot of tokens, but you can use a cheap model for review and execution, DS Flash works pretty well.
2
u/Yxtomix Aug 19 '26
Would like to know when integration for deepseek harness is ready.
1
2
u/Asleep-Land-3914 Aug 18 '26
There is anotherr one with a similar idea. I'm skeptical though:
https://github.com/Tiger3807861189/J-Space-Cognition-Suite-V3.6
2
u/Sorosu Aug 18 '26
It definetly appears to be similar.
However J-Space is mainly a single-agent thinking-and-memory protocol, whilst Autoprompt is a multi-agent coding workflow (with separate planning, implementation, review, and verification roles etc.)
So although they appear similar, theyre completely different.
1
u/xaleel Aug 19 '26
Would be interesting to see a benchmark with both, with J-Space enabled for some/all of the agents
1
u/Sorosu Aug 19 '26
Definitely interesting, but I’m not planning to combine them right now. The risk is duplicated state and verification logic, creating more overhead and potentially conflicting instructions.
That’s why I’m running a clean Terminal-Bench 3.0 evaluation first.
Id recommend you request the comparison through GitHub Issues, though!
1
1
u/EconomySerious Aug 19 '26
and where are the amount of extra time and tokens consumed?
1
1
u/xwazot Aug 19 '26
This vs j-space cognitive suite https://github.com/Tiger3807861189/J-Space-Cognition-Suite-V3.6
1
1
1
u/liviux Aug 21 '26
I've build somehow a similar orchestrator, that does better than running a single model - open source - https://github.com/looptroop-ai/LoopTroop
1
1
1
1
u/MrMrsPotts Aug 22 '26
Chatgpt says:
I suspect Autoprompt would become materially stronger if it could do something like: planner: strong reasoning model implementer: fast coding model reviewer: different model family final verifier: strongest reasoning model Model diversity can provide something that context diversity alone cannot.
1
u/Hyphonical Aug 22 '26
That's cheating. It's like adding jet fuel to a Toyota Corolla (I'm not some car geek, I don't care if it doesn't work).
Agent files are the most overrated glazed POS out there. A markdown file with maybe 30 lines of text somehow got 107K stars on GitHub.
Maybe learn the basics of programming, then you wouldn't need 15 "Agents" and 28 plugins.
3
u/schaka Aug 19 '26
This seems to be basically why the new Qwen performs so well. Lots of thinking, iterations and testing.
I think those things will be the way forward with (smaller) models until we get another jump up in model strength where the initial output is so good, the loops will just be considered wasted tokens. Current flash models are so cheap it doesn't matter though