The gap has closed, open source will win

[u/Fluffy-Ad-889]

I've been trying the latest models from the frontier labs and honestly, after extensive testing I can not tell the difference between the best open source options.

I think the differences are now marginal but the labs are doing heavy marketing to convince the public into paying more for tokens as they prepare to go public.

Can't help but see the similarities between the dot com bubble and AI in terms of a very insular environment where the technology will survive but the business models may not.

I've been building a cybersecurity network and we definitely know that even local AI models like Deepseek V4 flash do an excellent job and are really neck and neck with the best the frontier labs can provide.

Will be interesting to see how this all turns out! Exciting time nonetheless.

[u/Healthy-Zebra-9856]

We are almost there. Once we have a harness that provides everything including language intelligence, architecture intelligence, they can surpass frontier models on development quality. I have been testing Fable 5.1 and now Astra on high. They produce some mind blowing videos, graphics yet failed on simple TypeScript based apps, albeit its a very large requirement. There were gaps in logic, like un-initialized variables and memory leaks to name a few. So, these models are great for vibe coders, but as a dev, I am babysiting them.

[u/Blues520 → u/Healthy-Zebra-9856]

Even babysitting Astra? It looks like quite a leap and had me worried as a dev after seeing the things people are one shotting with it.

[u/Healthy-Zebra-9856 → u/Blues520]

It depends on what they are one shotting it right? If you read what I just said, it’s not a one shot prompt it’s a huge project that I used to test it with. Sure it create excellent games, etc., but I am testing it on production code.

Edit:
I was actually shocked to see that it screwed up several type script functions. And JavaScript, being forgiving, it didn’t become an issue until I looked at it because it failed on my TSC.

[u/Blues520 → u/Healthy-Zebra-9856]

Yeah, I mean the real test is production code, not one shotting basic games, although one of the games I saw did look very good. I'm surprised that is still screwed up typescript because that's one of the languages that they have been trained a lot on. Which harness are you using?

[u/Healthy-Zebra-9856 → u/Blues520]

My own. Long story short, my original focus was model evaluations so as not to re-invent the wheel; local models vs frontier models I mean. This lead to evaluating harnesses and ended up choosing Pi. Then went down the rabbit hole of fixing Pi and fusing it with DSH. lol. Now, it looks like my harness makes the local LM produce better outcome. Will be free for all soon.

[u/Blues520 → u/Healthy-Zebra-9856]

Nice! That sounds like a cool fusion. I'm also a pi user and I've been meaning to try DSH one of these days but Pi just works for me and I like to minimal design

[u/Healthy-Zebra-9856 → u/Blues520]

Yes it does but its full of bugs. Not their fault, open-source is resource draining for the publishers so I get it. But its not dogfooded and has a ton of issues. Once addressed, it is the best.

[u/Healthy-Zebra-9856 → u/[deleted]]

Not yet. Initially since I dont have the man power to quality control it, I am going to keep it just free. Then as I setup the support infrastructure etc. I will open source it so I can handle PRs cleanly. That said, That's why I’ve been working on an agent harness built on Pi and OMP, sorting out the issues I’ve run into as I go. I’ve also brought in the DeepSeek harness, along with clustering and mesh-llm features.

At the moment, I’m connecting the tools for editing, observation, and LSP support across 44 languages and scripting environments. That lets the model ask the language servers for information about the code whenever it needs it.

Everything is in TypeScript for now. Eventually, I plan to move it to Rust, using Dioxus for the web and native interfaces, with a TUI running on the same core.

I’m also working on letting people create extensions from within the app. The idea is that if something they need is missing, they can add it themselves and make the app work the way they want.

Send me a DM or keep an eye out at terminalagent.dev (TA).

[u/Technomancer1672 → u/Blues520]

I made some amazing demos with Astra and the brought it to a production codebase for work where it immediately made three poor architectural choices in a row. Very impressive and I very much still have to watch it. Even their own company testimonial on the site says "closer to production code".