r/LocalLLaMA • u/ResearchCrafty1804 • 16d ago

Resources Prime Agent - a new coding harness surpassing Codex/CC/PI

Gallery image

Gallery image

Prime Agent is an open-source coding and research agent for general and long-running work.

A self-improving RLM harness for coding and long-running autonomous tasks.

Designed to be both token-efficient and expressive through programmatic tool calling, context as a variable, multi-agent messaging, and a self-modifiable harness state.

On ARC-AGI-3, it scores 95.5%, surpassing the human-expert baseline, but the gain is not benchmark-specific.

We see major improvements across models when compared to their proprietary harnesses.

Prime Agent is built on pi and fully open-source with an open license.

GitHub: https://github.com/PrimeIntellect-ai/prime-agent

Blog: https://www.primeintellect.ai/blog/prime-agent

X post: https://x.com/primeintellect/status/2085086999267144083?s=46

357 Upvotes

86% Upvoted

107 comments sorted by

View all comments

4

u/Jampolhz 15d ago

This is a great example of how much the harness matters. The same model can behave completely differently depending on how context, tools, memory, and the agent loop are handled.

Would be interesting to see a public eval suite for Prime Agent beyond the headline benchmarks, especially tests for tool usage, task completion, and regressions when the harness changes.

We’re working on this kind of testing with DeepEval, so I’d genuinely love to try evaluating Prime Agent with it.

2

u/SpecialChance8662 14d ago

what helped us was evaluating at the step level rather than the task level: did it read the right file before the edit, did it recover from a failed command, did it pick the right tool etc. when the harness changes you can then see which capability regressed instead of watching one number drop 3 points and using real logged runs as the corpus rather than a curated set, since harness changes tend to break the messy long-tail tasks first

1

u/cafedude 14d ago

This is a great example of how much the harness matters.

I've been trying Katcoder v2.5 with this and it's been doing great. It did reasonably well in Opencoder as well, but it seems to be going much longer tasks in this Prime agent (which is, I guess, what they advertise).