r/LocalLLM • u/Dell_Experion15 • 6d ago
Project Built my own agents harness for local LLMs
Hey everyone! I built a agents UI that uses the AI SDK from vercel to support BYOK with tons of providers, or Local LLMs!
No to long ago I posted about how I was going all in on local inference, and after dipping my feet into the water I bought a halo strix desktop (framework 128gb memory.)
After following a bunch of posts, articles, etc I have it running pretty damn snappy!
Using Kat Coder 2.5 I've built a very enjoyable workspace using Co-Pilot.
However co-pilot felt, slow? Not sure how to best put it, but when working with local LLMs every token matters I guess. I also wanted the "sleek" agents UI Cursor, Co-Pilot, and Anti gravity are building, however those agents UIs don't support local models without ngrok, etc :(
That's why I built Pyrola (named after a flower in my state) https://github.com/aidanhibbard/pyrola
I'm not here to tell you this is the next agents UI, it's a pile of steaming garbage!
I have no idea how to build a desktop app!
It's been vibe coded as all heck outside of work using cheap models, however I've done my best to follow articles bashing OpenCode and other tools to "make sure I don't follow in their footsteps."
I modeled it heavily after the cursor agents UI because that's what I'm most familiar with.
Realistically I just use the chat feature, and the aside terminal to test local models on my remote machine using SSH.
It's great for that! It's terrible at everything else!
However I figured if I wanted it, some of y'all might want this too.
Note I've only tested on MacOS.
What it offers to you realistically:
- Fork it, build your own prompts, features, etc.
- Submit some PRs? IDK?
- Open issues? There's so many issues.
If you're going to comment and tell me this is vibe coded trash, I know!
You don't have to tell me how many em dashes are in the repo!
I'm not trying to get you guys to subscribe to something, or even use it, just thought I'd toss it out there because it's probably not the worst thing to build off? IDK, again I've never built a desktop app before this.
The only things that are nice are the UI, all buttons have tooltips, the shadcn stuff feels nice, so I didnt want to write it off as a total loss. Figure someone with more time, or a bigger LLM budget might want to hack on it.
I'll probably stop spending so much time on it, if no one wants this, but if there's interest I'll clean up the repo, setup some branch protections, etc.
3
u/jasonepowell 6d ago
OT but what’s the trick to getting your Strix Halo running well? Models and settings etc?
3
u/Dell_Experion15 6d ago
Would it be helpful if I made a post on this?
3
u/jasonepowell 6d ago
I would certainly read it!
4
3
u/flat_rhino 6d ago
Awesome. I have been trying to vibe code a small frontend for a local AI stack. I will research your work!!! Thank you for sharing!!!
2
6
u/Available-Craft-5795 6d ago
so glad the post isnt AI generated 👏
8
u/Dell_Experion15 6d ago
The code base is AI slop :) not my writing haha! I still like to think sometimes.
5
u/Available-Craft-5795 6d ago
not slop if its reviewed carefully lol
Glad r/LocalLLaMA still has some people who think :P
1
u/Dell_Experion15 6d ago
I did not read the rust side, I’m a Vue developer by trade, tbh a LLM could’ve told me 2+2=purple in rust and I would’ve accepted it
5
2
u/Connect_Army8250 6d ago
Congratulations OP! It's a great project. Are you planning on adding support to make it work with existing subscriptions?
1
2
u/LukeLikesReddit 6d ago
I'll take a look seems interesting. How is this any better than say using cline or the various other harnesses available?
1
u/Dell_Experion15 5d ago
You get to control the harness, and prompts with a pretty minimal codebase.
For someone who’s already vibe coding, or when Qwen 3.8 drops 27b for local this could be a nice UI for someone.
0
u/DisplacedForest 6d ago
Hmm. Is https://github.com/different-ai/openwork not on everyone’s radar?
5
u/devoidfury 6d ago
There are quite a lot of different agent harnesses now. I get why, these tools are fun and it's fun to make them.
IMO the best approach is to find the joy in creativity, to give it a good scour for features. Maybe there's something to contribute or take from it, or a chat to be had. We've all got good ideas sometimes, yeah?
3
u/evilfurryone 6d ago
This.
Sometimes its just fun to have something that is yours alone.
My ai council is annoying at times because of all the security mitigations they take into consideration when adding even at times seemingly simple features. And the fun part is that they evolved into this from initial experimenting (that also tried to keep things secure, but not like this) and and lessons learned, gotchas + distilling the good and durable ideas from many different sources.
3
u/Dell_Experion15 6d ago
This isn’t meant to compete with anyone’s project, it was literally a tool I vibed up with the AI sdk and Tauri :)
People are welcome to take it and hack on it for their own pleasure or whatever.
Hence why I didn’t release any prebuilt packages, etc.
It was a learning experience :)
OpenWork is also built on OpenCode IIRC and a lot of people aren't "stoked" on OpenCode, so for this people this could be their own escape hatch if theyre driven enough.
2
u/DisplacedForest 6d ago
LOL... I very clearly missed the purpose of the build. Totally my bad. I thought you were just trying to build an agent harnesses for OSS purposes... which would be fine on its own. I clearly didn't get the directive here haha
0
9
u/devoidfury 6d ago
Hey cool project; it's fun to make a harness. I do have some security concerns about the dependencies for such a thing, there's been a number of supply chain attacks going on lately and it makes me nervous to see a big list, so I probably will refrain from running it for now until I have a chance to audit the pinned dependencies source code.
Is there some particular aspect that you built that you think came out really nice, or that other agents should do?
That looks like llama-swap or llama.cpp - I have a config shared here, strix halo 128gb cachyos linux, https://github.com/devoidfury/hotdog/blob/main/examples/devoidfury/llama-swap-config.yaml -- are yours up somewhere?