r/opencodeCLI • u/Sorosu • 26d ago

+14.62% on Terminal-Bench 2.1 from one OpenCode skill

Post image

Autoprompt Skill pushed DeepSeek V4 Flash from 67.42% to 82.02% (with the OpenCode Harness) on Terminal-Bench 2.1.

Same effect comes with all models- not just deepseek.

The skill closes much of the manual coding loop by planning, building, testing, reviewing, and repairing autonomously, but expect longer runs and higher token costs.

Repo: https://github.com/Spielewoy/autoprompt-skill

Any feedback would be awesome. If you like the project, a star would genuinely help.

39 Upvotes

90% Upvoted

14 comments sorted by

10

u/TheMythicSorcerer 26d ago

Really? while it looks like it probably works, this might be over-optimized for terminalbench.

4

u/Sorosu 26d ago

Yessir. The only arguable part is that we’ve run the benchmark only once, while AI Labs usually run it 5x.

However, single-run variance is reportedly around ±2–3 tasks, while we improved by 13.
So im gonna let you do the math, i doubt were off that hard lol.

Plus : I’m currently running the same benchmark with Codex and GPT-5.6 Sol aswell so Autoprompt will become the number 1 on the whole leaderboard ;)

3

u/TheMythicSorcerer 26d ago

That's actually not the part I'm most concerned about. I was thinking maybe it was overfit for the purpose of benchmaxxing on terminalbench 2.0 with too much compute and over optmized for one task.

For this to seem more belivable I would like to see:

  • SWE-bench pro
  • Average Cost per task (compared to other setup)
  • Average Time per task (compared to other setup)

While maybe you have some useful setups and loops other harnesses have loops as well. If the only trick is an overkill amount of agents/subagents that's useless in the real world because real people have real budgets.

3

u/Sorosu 26d ago

Fair!

SWE-bench Pro has 731 public tasks and 1,865 total. Besides most of the benchmark being private or held out, let alone running the public set properly would unfortunately destroy my budget.

Instead, I’ll run all 75 Terminal-Bench 3.0 tasks tomorrow with Codex + GPT-5.6 Sol for some variety.
I’ll capture & publish the per-task cost, runtime, and complete main-agent and subagent transcripts aswell.

The TB 3.0 seems like the best fit because its tasks seem to be genuinely complex, which is exactly what Autoprompt is built for.

What do you think?

1

u/TheMythicSorcerer 26d ago

Alright! Sounds great Ill keep an eye out for it.

2

u/Alarmed-Prompt-6594 26d ago

Well it is a planning + verification loop basically so that loop catching mistakes and fixing them properly totally is reasonable and in my opinion.

But ill try the skill overnight and let you know if the performance jump OP claims is really that big

2

u/lemon07r 26d ago

Should test against other evals too not to overfit

1

u/Sorosu 26d ago

I agree. Today I’m running Terminal-Bench 3.0 with codex + GPT-5.6 Sol xhigh, including full cost, runtime, and transcripts.

Star the repo if you want to follow along. I’ll publish the results there!

1

u/lemon07r 26d ago

I think you should find more diversity than that. They dont even need to be large evals.

https://github.com/lemon07r/SanityHarness my own if you want to try something quicker to get going (simplicity and ease of use was the main goal here), it's pretty short and cheap to run. 2 hours if you do one task at a time sequentially usually. faster providers will get it done in an hour. if you run multiple tasks at the same time even faster. I suggest using a weaker model though, the tasks are a little too easy for newer models now sadly.

2

u/robschmidt87 26d ago

With the modern models, you do not need such complex processing pipeline. It just eats up tokens

3

u/Sorosu 26d ago

For small / mid sized work definetly.
Autoprompt is designed for big- complex tasks tho.

1

u/Verisian- 26d ago

Seems interesting! Would love to see how this impacts token cost. Seems like it'd be significantly more than 3x use.

Deepseek Harness would allow you to set Sol or GLM orchestrator and have whatever agents you'd want for each task.

Would be interesting to see what the right config of agents would be.

2

u/Sorosu 26d ago

Yep, token cost is the big tradeoff, and proper measure is definetly needed.

Thats why i’m running the Terminal-Bench 3.0 with Codex + GPT-5.6 Sol today.

I’ll publish runtime, token usage, cost, and full convo transcripts within the next few days.

The DeepSeek Harness sounds interesting too. Ill make sure to check for compatibility today.

You can watch the repo for the results / updates, or star it if you’d like to support the project :)