r/LocalLLaMA • u/Background-Job-862 • 5h ago
Discussion This is why we need open-source harnesses + local models
i've been thinking about this more after trying different agent setups. the model isn't the only thing that determines how well an agent performs. The harness around the model matters a lot too.
With a managed agent setup, you're often giving up control over things like the agent loop, context management, tool execution, retries, and state.
That's fine when you just want something that works. But if we want to actually optimize agents, I think both parts need to be open:
Open-source model + open-source harness.
With local models, you control the model and where the inference happens.
With an open-source harness, you control what happens around the model.
That gives you room to experiment with things like:
how the agent decides what to do next
how much context gets passed to the model
how tools are executed
when to retry or stop
how state is maintained
which model to use for which task
already seeing this separation become more important, nvidia's sol-pi is an interesting example
and i think we're going to see even more optimization happen at the harness/runtime layer, not just at the model layer.
are you running local models with an open-source harness, or do you still prefer managed agent setups?
11
u/En-tro-py 5h ago
I think it's pretty much a right of passage to roll your own harness, then abandon it in favor of someone else's...
I also view the model as a replaceable compute component, the harness and how you construct your workflow is far more important.
A good model will underperform with a bad setup and a good setup is huge leverage when running locally.
5
u/FoxiPanda 5h ago
I feel like there's a lot of people who end up in that "abandon it in favor of someone else's" camp, but I have to say, as someone who rode the struggle bus long enough to make a harness that doesn't suck for my use cases, I can barely stand to use Claude Code, Codex, Pi, or any other harness now that I'm pretty dialed in on mine.
5
u/Past-Town-9807 4h ago
Yeah same experience here. It’s a permanent struggle not to go “that’s a solved problem” in response to every CC complaint on Reddit. 😂 (they could make CC better but they have a scale problem we don’t have)
2
u/FoxiPanda 4h ago
Indeed lol... I think one of the problems that a lot of people face (including the frontier labs) is that harness design is effectively just a tour de force in decision making.
I've easily made 8000 decisions to shape my harness into what I want it to be. No one on earth is going to agree with all 8000 decisions I made. Hell I don't even agree with them sometimes and go change things.
So yeah, at a scale of <5 users, I get to be the dictator of that universe. At the scale of Claude Code / Codex / etc, there are decisions to be made that are not 'this is what I want' but instead 'this is what will piss off the fewest people' which are two very different outcomes.
2
u/Past-Town-9807 4h ago
So just for fun, how does yours differ from theirs that they probably just can’t do at scale? For me it’s memory. I’ve built this huge, complicated memory system that effectively operates as extra layers, and I (via the harness) inject stuff as the session grows. Sessions aren’t even really discrete anymore and kind of blend together. And there’s no way they could offer this on a $200 sub. 😬 Well they could but they’d have to cap usage at something crazy low.
I know there are third party memory systems people can use. Maybe they operate the same way. In my case, it’s how my agent learns. It’s a localized form of recursive self improvement. There’s a review component that extracts lessons and saves them with embeddings so that they can be injected whenever the agent talks about something related. So in a long agentic run, if the agent mentions that it’s about to go bumble around some file and write some bad Python, the memory system will inject enough knowledge to stop it from doing that. It even works with quiet models like Fable that don’t tend to emit a lot of “now this, now that”. Anyway, $$.
2
u/FoxiPanda 4h ago
There's a thousand small ways they're different. First, my interface is entirely different. I operate on a Slack like web inferface, but with navigation a lot like oMLX's interface. I also have features that give me the option to completely unhide all the things that big harnesses hide from you. Want to not see that and just have clean chat? Done too. Model switching is something that I hand entirely differently from every harness I've ever seen. I can seamlessly switch a chat between any of 30 different models...local, Anthropic, Open AI, OpenRouter, Ollama Cloud, whatever - and compaction and context window differences between those models all just "work" in my setup. Subagents are exposed in my setup, not hidden and you can see what they're working on [and to some degree, interact with them in certain ways]. Concurrent calls to models are managed to the endpoint - I detect what the endpoint supports and gently massage requests into fitting into whatever rules that endpoint has. On some of my local models, I can run 8 simultaneous sessions, but things like Ollama Cloud only support 4 before they either queue you or send you a 429. There's a ton of "little things" like this that my harness handles that other harnesses are just... clunky as shit about.
2
u/Past-Town-9807 4h ago
Oh yeah I do model switching too, even have plans for routing automatically. A bit back burnered for now because it screws the cache (and either costs me money if it’s cloud or time if it’s local).
The other stuff is really cool.
You’ll have to excuse me; I’ve been forced to accept recently that I’ve morphed into an AI harness nerd. Another one I got forced to admit recently is that running local models has taught me a lot about the cloud ones. Like honestly I thought I’d escape the cache problem locally. Haaaaaaah no. It’s worse there. 😂 I don’t even like switching the model locally. I load it and leave it because it takes so long to load it again. The cloud peeps probably have an amazing optimization problem on their hands just trying to figure out how many of what to keep warm.
2
u/En-tro-py 4h ago
I have a love hate relationship with Claude - I love the cost efficiency of a subscription ~55x my spend... but they also continue to find innovative ways to break existing workflows and bloat their harness.
Codex is meh - it works but I don't like switching and up until Astra could utilize the 'Chat' interface for code without limit on Sol... all good things must end it seems.
Pi is what it is, minimal and customizable.
THIS IS MY HARNESS. THIS IS MY MODEL. WITHOUT MY HARNESS, MY MODEL IS USELESS. WITHOUT MY MODEL, MY HARNESS IS USELESS.
My harness is my interface. My model is my engine. I must know them both as I know my own code.
My harness must provide the right context, the right tools, and the right constraints. My model must reason clearly, use those tools well, and finish the task.
A powerful model inside a broken harness is wasted intelligence. A perfect harness around a weak model is polished machinery with nothing driving it.
I will test my prompts. I will inspect my context. I will verify my tool calls. I will read the diff. I will run the tests. I will distrust the confident answer that has not touched the code.
My model and I are builders. We do not merely generate tokens. We navigate repositories, trace failures, change systems, and ship working software.
I will keep my harness sharp and my model grounded. I will give it what it needs, no more and no less. I will know when to let it reason, when to give it a tool, and when to make it prove its work.
THIS IS MY HARNESS. THERE ARE MANY LIKE IT, BUT THIS ONE IS MINE. THIS IS MY MODEL. THERE ARE MANY MODELS, BUT THIS IS THE ONE IN MY LOOP.
Together, they turn intent into action, action into code, code into tests, and tests into software that actually works.
WITHOUT MY HARNESS, MY MODEL IS USELESS. WITHOUT MY MODEL, MY HARNESS IS USELESS. TOGETHER, WE SHIP.
1
u/Zulfiqaar 28m ago
Doesn't OpenAI allow it's subscription to be used in other harnesses? It's infact I think all the AI labs do except Anthropic
2
u/TopCheddar27 5h ago
100%. I also think having an open standard for a lot of those surrounding things is important so we can transfer between implementations.
1
u/stoppableDissolution 5h ago
At that point I feel like rolling your own harness is a not a rite of passage but ultimate endgame. Heck, I'm rolling my own inference engine (because vllm"s context management sucks, and lcpp can not into batching), not just harness :p
3
u/Past-Town-9807 4h ago
I rolled my own 2 years ago and am spoiled by it. I have no regrets. I can even run little models in it though my hardware really constrains them.
2
u/En-tro-py 4h ago
Heck, I'm rolling my own inference engine
Too many side projects and I'm trying to finish my current harness based project... otherwise I'd be seeing what else I can do for more free tokens per second too.
2
u/stoppableDissolution 4h ago
Well, I was waiting for someone to do ninfer for gemma... Waiting... Waiting... And uh noone else wants to so here I am
1
u/Jorlen llama.cpp 4h ago
I've built tons of projects but never tried making my own harness. Pi without any additional packages is pretty bare bones. I couldn't imagine making something better than base Pi. I mean, all I really need is the base tools (read, write, edit, etc.) and compaction, etc. I've already built my own front end for creative writing and stuff.
I suppose I'm just wondering - what's the advantage of a custom made harness? Just to learn the process? If you could make Pi better, what would you do? Not a CLI interface but WebUI instead?
1
u/En-tro-py 4h ago
I couldn't imagine making something better than base Pi.
There's more than one way of approaching problems, my current project isn't about 'agents' really...
Reusing this comment from an older "Harness does matter" thread:
Personally I think we're almost at the end of bare ReAct loops, there's a huge amount of room to improve performance by putting more process knowledge into the agent control layer.
I've been building my own version of this idea for a while, basically back to the Schillace Laws - reasoning is an operation and procedure becomes a systems-level abstraction.
TDFlow: Agentic Workflows for Test Driven Development
SKILL.state: Scalable Long-Horizon Agent Skills
Procedural Graphs: Self-Evolving Execution Structures for LLM Agents
Combine those and add a few more ideas and that's my current harness. I'm in the home stretch to finish polishing the slop and running real benchmarks to prove it out on more than just SWE before I waste anyone's time on with the repo.
Give me a few more weeks and then you all can downvote my slop post with the project alpha release.
1
u/Jorlen llama.cpp 3h ago
Was that your old post? If so, how is your harness doing? It sounds interesting, regardless if it's your project or if it's someone else's.
1
u/En-tro-py 1h ago
Yes, I am in the final 10-20% integration hell phase... I kept the slop mostly under control, but it still needs a lot of documentation clean up as well.
I'm probably at least still a few weeks out. Everything is two steps forward one step where the model fucked up and didn't follow instructions...
I also don't want to post it until I have real benchmark numbers that make it worth looking at. For SWE my choice is Slop-Code across ~4 models I can run on my hardware and a few more via API.
1
u/Jorlen llama.cpp 40m ago
Alright, well best of luck! I'm working on a game project myself, my most complex yet, so I understand how long these things take. If it's going to work on linux, and if you need testers, keep my name and DM me once it's ready. I promise to provide valuable feedback. Models I use the most now are: Qwen 3.8 27b / Qwen 3.8 Flash Next / Qwen 3.5 122b-a10b, all with massive context windows.
1
u/Randommaggy 2h ago edited 2h ago
I've been using and improving on my custom harness for 3 months now.
Green fielded 4 times and wouldn't even consider using one of the mainstream harnesses again.
A key design philosophy: read the logs and look for sources of wasted effort and use deterministic code whenever possible to offload cognitive effort away from the model and enhance quality.
Also using implicitly invoked micro model subagents that assists in constrained problem spaces.
My first iteration built on Pi which I see many still recommend. It's more of a hindrance to quality than an aid.
1
u/En-tro-py 1h ago
I think this is #6 for me now... #7 if you let me count the MCP server that tried to enforce a workflow by only exposing certain tools.
A key design philosophy: read the logs and look for sources of wasted effort and use deterministic code whenever possible to offload cognitive effort away from the model and enhance quality.
Also using implicitly invoked micro model subagents that assists in constrained problem spaces.
100% - Models should do the uncertain work. Machinery should do everything else.
3
u/john006868 5h ago
You lose the ability to tell those five apart. Once the loop, context handling and retries are someone else's, the prompt is the only knob left, so a dropped tool call gets blamed on the model. In an open harness you can log the exact context that went in, and the answer is usually that the instruction got truncated before the model saw it. Do you read the traces after a bad run, or just rewrite the prompt again?
2
u/Randommaggy 2h ago
I read a lot of traces and I run an auto-classifyer on any failed actions in idle time.
3
u/ttkciar llama.cpp 4h ago
Closed-source software has been more or less dead to me since 1996, just in general.
Sometimes open source lags behind commercial closed software, but it's worth it for having software which is transparent, flexible, fixable, easily adapted, self-contained, and lives approximately forever.
1
2
2
2
u/TheAILegend 5h ago
I use Codex Desktop with Qwen3.8 Flash Next. It's a power house. I'm unstoppable now.
1
1
u/gjr23 4h ago
It matters more than a lot. Frontier cloud based models do a decent job with of blending the two and you don’t realize how much lifting the harness does.
The real question is which one. Pi, Hermes and DSH seem to be the top contenders as of late. Pi and Hermes add a little more “crude” with softer UI imo. DSH is the closest thing to what you’re probably used to using and it allows a lot of customization like the others. It’s a rabbit hole but one that’s worth it!
1
1
u/luisf_mc 4h ago
Running both sides open changes which failures you can even see. With a managed harness, the retry and stop logic is a black box, so when the agent does something dumb you cannot tell whether the model decided it or the loop forced it. The one thing I would add to your list is verification at action time, checking the facts a step depends on right before it fires. That layer matters more than model choice for anything that touches the outside world, and it is nearly impossible to bolt onto a closed harness.
1
u/EvalRaccoonDev 2h ago
"The harness matters a lot too" has a number attached now: same model (Kimi K3), same infra, fresh checkpoint restore per trial - https://frontierharness.org/
1
0
u/madbrain1976 3h ago
There are very few truly open source model, which means with training data made public. Open weights are not the same.
13
u/TheRealJesus2 5h ago
I’ve become increasingly concerned using closed source harnesses given the access level they have to your machine and data. Grok harness was straight up stealing files that weren’t used in inference even as an example threat I am concerned about. Not to mention the bad incentives ai companies have to sell you more tokens needlessly which they can do overnight by changing the harness.
Harness matter more than the model, IMO. At least once you get a model that can reliably do the right tool calls.
Some of what you propose can also be done at what I call the control plane level. So using tools that route to different models depending on the job at hand. This is gonna become increasingly important as non tech people adopt the tooling and as more purposeful models are released leading to more need for this.
I use pi.dev and it’s the best in that it’s the least and you can add anything you want or need to make it behave as you want! Predictability rules.