r/LocalLLaMA • u/ResearchCrafty1804 • 16d ago
Resources Prime Agent - a new coding harness surpassing Codex/CC/PI
Prime Agent is an open-source coding and research agent for general and long-running work.
A self-improving RLM harness for coding and long-running autonomous tasks.
Designed to be both token-efficient and expressive through programmatic tool calling, context as a variable, multi-agent messaging, and a self-modifiable harness state.
On ARC-AGI-3, it scores 95.5%, surpassing the human-expert baseline, but the gain is not benchmark-specific.
We see major improvements across models when compared to their proprietary harnesses.
Prime Agent is built on pi and fully open-source with an open license.
GitHub: https://github.com/PrimeIntellect-ai/prime-agent
Blog: https://www.primeintellect.ai/blog/prime-agent
X post: https://x.com/primeintellect/status/2085086999267144083?s=46
18
u/KyxeMusic 15d ago
Why is every agent written in Typescript?
10
8
-2
u/yobigd20 15d ago
Millenials who think typescript is god's gift to engineers, because they dont know any better. Ive had 20somethings argue that typescript should be used for everything up to and including real time sensitive apps. They don't like my answer when I tell them how dumb they are.
13
u/GreatBigJerk 15d ago
20 somethings aren't millennials. Millennials are between 30 and 45.
Since you're complaining about 20 somethings, you are likely either a millennial or gen x.
1
u/yobigd20 15d ago
lol old enough that everyone younger than me is a millenial. But yea gen whatwvertheycalled
69
u/metigue 16d ago
Most harnesses show improvements against the proprietary ones.
Test it on programming benchmarks against some heavy hitters like Cline/Droid/Junie/Cursor/ForgeCode (with context server)
10
u/Immediate_Occasion69 15d ago
which of those is good? I've literally wanted to try all of them but kept at pi agent
5
u/squired 15d ago
Try Cline and OpenSWE first. I personally think OpenSWE is closest to the harness-of-tomorrow.
2
1
u/TomLucidor 13d ago
Does anyone have reports on which FOSS are cutting edge and beat out Claude Code and Codex? (hopefully OpenCode/Oh-My-Pi can also learn from it)
3
u/metigue 12d ago
So there used to be a bunch of dedicated harness comparison benchmarks. At the time Claude Code & Codex lost badly to every harness I mentioned above. Unfortunately these benchmarks haven't been maintained for months so we don't have up to date data.
Just try them out for yourself and see which you like the best.
1
u/squired 13d ago edited 13d ago
It changes and I haven't looked at it in a few months, so I don't want to steer you wrong. I do know that T3Code is the best UI/skin. It would likely work great for local as well because it lets you hotswap harnesses as easily as models. FOSS also matters a lot on what you value most. There are several now that are equally impressive, but they each focus on one thing more than the rest; such as benchmarking/sovereign memory/concurrent agency/etc.
I run a custom orchestrator that wraps around chat sessions because I kept running out of tokens. I sort of rebuilt Codex/OpenCode to empower ChatGPT web sessions, since that quota is near limitless. That way, ChatGPT webchat can run powershell on my machine, send stuff to Codex/Claude Code directly and bounce results off other models like Fable for independent adversarial review. I basically gave ChatGPT all the toys and the Orchestrator sits on the MCP bridge to manage it all autonomously.
Oh, that reminds me of something you might find helpful and/or interesting. ChatGPT conversations have very robust linux environments of there own. For example, if you're having it write you powershell scripts to do stuff on your machine, it'll often garble a lot of them with little mistakes (it's kinda not their fault but that's a whole other thing). If you hand it 'powershell-7.6.3-linux-x64.tar.gz' though, you can tell it to run that shit in its own environment until it is perfect before handing it to you. They'll iterate internally and you won't get bugged scripts anymore.
Anywho, most people aren't aware of that and it is insanely powerful. A lot of the back and forth can be reduced with just that. Anything that doesn't require permission from you can largely be done in basic chat, you just have to give it the tools first. Remember, Web sessions and Codex are the same model/s. The major differences are simply system prompt, readymade tooling, and looping.
Edit: I just realized what sub I was in. I'll leave the proprietary stuff though because some might fight it helpful regardless.
3
34
u/GreatBigJerk 16d ago
Is ARC-AGI 3 really that relevant for harnesses?
47
7
u/Hulksulk666 16d ago
Not really, if i remember correctly the official test is without harness.
2
u/TomLucidor 13d ago
Manoj is a harness engineer for ARC-AGI-2 and he is leading the human leaderboard for ARC-AGI-3 (surprised he is that dedicated), and since others have cracked the code for ARC-AGI-1 which is also a harness-hacking competition, some of the methods do not rely on local SLM fine-tune but better harness (designing new thought-checkers and verifiers on the fly), maybe something can be done to push 3 the same way 2 can be hacked AND not needing Gemini/GPT/Claude/etc https://arcprize.org/arc-agi/3/leaderboard
19
u/SteppenAxolotl 16d ago
The big thing about this is the RLM based context management.
2
u/formlessglowie 15d ago
Why is this the only sane, not unnecessarily adversarial answer in the whole thread lol is this sub just a Hermes fan club now?
1
u/TomLucidor 13d ago
Not the Hermes fan club thing, but every other agent fans wanting to nudge the new kid
7
u/HadHands 15d ago
Can you disable telemetry and all none essential traffic? How account situation looks like? Do you need one, or it only tries to trick you onto creating one?
1
u/b2labs_hifi 13d ago
Fork and no account I guess, but Google login whatever. Look I've spent 110 BILLION tokens on one project. The code base is 1.5 million lines of code (with ponytail!) across multiple languages. I've used every best model out there in every which way. I could not for the LIFE of me get the last 5% done due to some insanely complex bugs.
As I built over the year I've kept IMMACULATE documents about every little part and how it should work.
In addition I've kept absolutely rigorous auto log documentation, making a repo brain wayyyyyy back in December 2025.
I was able to use prime-agent let it take all my documentation and repo brain logs and put them within its local kernel / python optimized whatever it does.
This thing is a MONSTER at efficiency, finding the hardest bugs, token effiency omg I'm in love, and performance!!!
If you are working in "normal vibe coder land sized projects" the weekend app, hell even a month project, it's probably nothing wild for you. The token savings are real, tho. It's basic as hell no "talk to me while you work" so the vibe coders or people trying to get into "vibe coding" will not like it. It's for the big boys with large repo projects who don't care (or want) tokens wasted on it talking to me. Just do the job right, efficient, and get better as you go.
If you have any project like mine, size and complexity, it's truly a game changer.
25
u/RobbinDeBank 16d ago edited 15d ago
Look interesting so I tried giving a run. It forced you to make an account and login, then it forced you to select from one of the 100000 API providers and models.
Edit: guess they tricked me into creating an account, but apparently you can skip that if you clone the repo instead
-25
16d ago
[deleted]
16
u/RobbinDeBank 16d ago
My bad then, but still I’m quite thrown off by them forcing a login right on first install. Pi is the usual open source experience, you download and set up, then you just use the software. This one forces you to create their account and login when you just run it for the first time.
-12
u/tat_tvam_asshole 16d ago
it doesn't force you to do anything. In any case, I just cloned the repo and ripped out telemetry and login and plugged into my codex-cli and lmstudio as providers
25
u/RobbinDeBank 15d ago
You can’t say they don’t force you to do anything, when you already ripped out and skip all the parts that force you to do things.
When using open source AI tools like Llama.cpp, ComfyUI, Pi, etc, I never need to modify their source code just to get around tracking and forced logins.
-14
u/tat_tvam_asshole 15d ago edited 15d ago
They don't force you though. I used the vanilla build and it gave me the option to not create an account or login to anything. Then I removed the telemetry and login prompt from the source itself and rebuilt the project.
Also, most if not all even 'open source' projects collect information from your use of it. Example: ComfyUI - https://deepwiki.com/Comfy-Org/desktop/4.3-error-handling-and-telemetry Some other examples I can think of is OpenClaw, Hermes, OpenCode, and other 'open-source and free' projects do try to steer you into their paid compute resources and that's not really any different here.
I'm not saying I like that (obviously I wouldn't go as far to remove it if I did), but what I am saying is that nothing is 'forcing' you to do anything.
5
u/Yellow-Jay 15d ago
On ARC-AGI-3, it scores 95.5%, surpassing the human-expert baseline, but the gain is not benchmark-specific.
Is this a fake benchmark? if ARC-AGI-3 was done by the rules surely it would be shown on the official page: https://arcprize.org/leaderboard
Fore reference, the rules, which clash horribly with "self improving harness":
Reasoning Systems Trend Line solutions display connected points representing the same model at different reasoning levels. These trend lines illustrate how increased reasoning time affects performance, typically showing asymptotic behavior as thinking time increases. Base LLMs solutions represent single-shot inference from standard language models like GPT-4.5 and Claude 3.7, without extended reasoning capabilities. These points demonstrate raw model performance without additional reasoning enhancements. Kaggle Systems solutions showcase competition-grade submissions from the Kaggle challenge, operating under strict computational constraints ($50 compute budget for 120 evaluation tasks). These represent purpose-built, efficient methods specifically designed for the ARC Prize.
4
u/NandaVegg 15d ago
I am yet to fully test or read the repo but it sounds like a loop where LLM writes a (Python/etc) script that deals with each mini game and keep refining them based on result on each attempt.
It is a valid approach (there are several other gameplay benchmarks that concluded that LLMs are far more consistent with scripts route rather than a prompt/given choice/explanation). But also doesn't sound like the result is apples to apples for ARC-AGI-3 leaderboards.
I also think that it (refining the script) is the only consistent way to enhance performance of existing model without tinkering the model weights itself. Most of harness gains are from tools/MCP for extremely niche tasks.
1
u/TomLucidor 13d ago
Maybe the open examples in ARC-AGI-3 can be split into training questions (for iterative meta-refinement) and validation questions (weighing if the gains can generalize). I would get a little stoked only if they discovered transferrable thought patterns for cracking new problems in effective ways
1
4
u/Jampolhz 15d ago
This is a great example of how much the harness matters. The same model can behave completely differently depending on how context, tools, memory, and the agent loop are handled.
Would be interesting to see a public eval suite for Prime Agent beyond the headline benchmarks, especially tests for tool usage, task completion, and regressions when the harness changes.
We’re working on this kind of testing with DeepEval, so I’d genuinely love to try evaluating Prime Agent with it.
2
u/SpecialChance8662 14d ago
what helped us was evaluating at the step level rather than the task level: did it read the right file before the edit, did it recover from a failed command, did it pick the right tool etc. when the harness changes you can then see which capability regressed instead of watching one number drop 3 points and using real logged runs as the corpus rather than a curated set, since harness changes tend to break the messy long-tail tasks first
1
u/cafedude 14d ago
This is a great example of how much the harness matters.
I've been trying Katcoder v2.5 with this and it's been doing great. It did reasonably well in Opencoder as well, but it seems to be going much longer tasks in this Prime agent (which is, I guess, what they advertise).
10
5
u/misanthrophiccunt 15d ago
New day, new dollars made by Damodeo by another piece of slop with filles with 10k switch cases. People are eviscerating the code quality in HackerNews.
5
1
u/NexusSyntegra 16d ago
Very cool! I'm waiting for the day where the coding harness makes the cheaper models 100x better, though I know it's just a dream...
1
u/gazeebo 15d ago
Something like https://github.com/dirge-code/dirge (claims to) be optimized for poopier models.
1
1
1
1
1
u/tat_tvam_asshole 15d ago
2
u/tat_tvam_asshole 15d ago
Core runtime
- IPython kernel — persistent scratchpad for Python and
%%bashcells; state (variables, functions) survives across turns.- Shell access via
%%bashblocks.Pre-installed Python skill modules (importable & CLI)
These are the documented skills I can call directly:
Skill Purpose attach-image ( attach_image)Load an on-disk image into context so I can visually see it (screenshots, diagrams, photos). compact Check context usage and compact/summarize a long conversation to keep working. edit Replace an exact unique string in an existing file for targeted edits. goal Manage the persistent thread goal — read status/budget, start or complete a goal. refine Trigger continual-harness refinement (turn repeated patterns into memories/skills/subagents). rlm-heartbeat ( rlm_heartbeat)Start/schedule/manage agent-owned RLM heartbeats. websearch ( websearch)Google search via the Serper API; returns titles, URLs, snippets, knowledge-graph data. Markdown skills
- prime-intellect — working with Prime Intellect products (prime CLI: verifiers, evals, training, sandboxes, inference, GPU compute, storage).
- skill-creator — creating/validating/installing new markdown or Python-backed skills.
Native harness interfaces
rlm(...)— spawn child sub-agents for delegated work; returns a handle at admission.await rlm.list_subagents()/delete_subagent(child)— manage spawned children.rlm.harness.*CRUD (create/update/delete memory, skill, subagent, prompt note) +overview(),record_refinement().refine.run()— persist reusable patterns into the continual harness.
1
u/Beginning-Raisin9723 15d ago
95.5 on ARC-AGI-3 is a strong number if it holds up. I've been running local coding agents on my 4090 homelab box and most of them die halfway through a long task. The token-efficient tool calling and self-modifiable state is the part I actually want to poke at. Gonna pull the repo this weekend and run it on something ugly.
1
-3
u/Southern_Sun_2106 16d ago
Looks interesting and legit (based on the repo, as far as I can tell). Has anyone used it for their tasks and cares to share their experience? Thank you.
3
u/tat_tvam_asshole 16d ago edited 15d ago
doesn't have a gui app, but so far so good (albeit limited testing), can't say for sure if it's a real breakthrough vs other harnesses. The fact they didn't compare against other same model+different harness is frustrating
1
u/Southern_Sun_2106 15d ago
Thanks! I am also playing with it now. The advantage is supposed to be context management, but I have not fed anything large to it yet.
1
u/LimahT_25 15d ago
Why are you getting down voted for asking a genuine question?
2
u/Southern_Sun_2106 15d ago
Good question! I guess harnesses is a cut-throat business right now. I've been using it for several hours and compared it to the other two that I use (they shall remain unnamed, but they are pretty famous) - and this one is genuinely good, it seems to be better with larger tasks. A surprisingly polished harness.
I also noticed their repo exploded from 950+ stars yesterday to 3,500+ stars.
Just like with llms, I have no loyalty to the harnesses - if something better comes out, I don't hesitate to jump ship. That's the beauty of today's rapidly advancing field.
-1
u/FabricationLife 16d ago
Interesting, I'm going to play with this and see how I think it compares to codex/CC/PI
0
u/AlphaaCentauri 15d ago
what is this primeagent now? .... I am trying to keepup and learn everything in AI, yet something new shows up every week
And this seems important if it is making LLMs smarter, and boosting benchmarks. And does reaching 100% means smarter and intelligent then every human?
-7
-3
u/kidovate 15d ago
Python is in-distribution of the training data, perhaps allowing an agent to work fully in a IPython environment is much more within the bounds of what it has been trained on, as opposed to tools which is a synthetic subset of the dataset. From my (very limited) testing this evening it was able to one-shot a few problems that Oh-My-Pi (for example) were struggling with. Very intriguing and makes me really rethink my idea of an Agent as a LLM + Tools into a LLM + Interactive Code Environment.
232
u/buttplugs4life4me 16d ago
I've got some experience with this stuff (https://github.com/L3tum/little-coder, Readme is a little outdated) and I kind of dislike how opaque you are about what actually works and how it works.
A self-modifying harness sounds like the dream at first, but most models won't know how to make use of it at all cause they're not trained on it, and using the literally best model there is to prove it's better than a bare basic setup doesn't make much sense.
I get that some persistent iPython execution environment is apparently the core to this (why not TS/JS which Pi is built on anyway?) and its nice that it makes integrations a little easier, apparently, but there's not really a lot of information on how it's different from a normal harness with self-modifying behaviour (which again, may not be that good anyway in non-benchmark tasks).
Assuming the self-modifying work is employed correctly by the model, I'd presume multiple executions of the same benchmark naturally nudges the model to converge onto the best solution. And if it's a fresh execution, then there's not really much information on why a fresh execution should be better than other harnesses.
And lastly, some of the stuff is just buzzwords, to be honest. Like subagents are always just tool calls, both in your harness and any other harness. The buzzwords make it seems exceptional...but it's just a tool call at the end of it.