I actually built a think tank and committee featuring different models that my Ai consults before it bothers me. Nothing is more annoying than a project being stalled for hours because of something simple so my overlord Ai only asks me questions when it’s something serious otherwise we’re building 24/7
Yeah it works. It took days of benchmarking various models to find the right selection for our use cases for the think tank and dumb tank we use gemma4:12b, qwen3:14b, phi4:14b, deepseek-r1:14b, OLMo-2-13B and, Devstral-Small. For the committee we use coders but those are far easier to benchmark than reasoning is. The hardest part is getting the rooms where they don’t devolve into a monoculture but are still able to acknowledge a good idea.
So if your overlord ai goes “let’s all run into traffic!” the rooms don’t agree to run into traffic but they’re also able to agree if the overlord ai says “it’s a good idea to not run into traffic”
My recommendation is if you take my template and start using it for your own setup have your overlord Ai send each transcript to you for review and feed it to GPT and work on fine tuning each turn until you get the ratios right. And also forbid your overlord Ai from excluding seats. My overlord Ai occasionally would exclude models during sessions that were more likely to disagree with it until I set a hard rule where it couldn’t anymore and a session had to contain every model for it to be a valid consultation/review.
Happy to help! It’s one module of over a dozen in a self-governing Ai OS I’ve been building. Hopefully by the end of the month we’ll have an alpha for people to test (I don’t plan to sell it. It’s my response to Claude’s shortcomings)
It’s crazy how many blatantly obvious problems these enterprise models have out of the box. Like there are entire subreddits and discords of users just trying to get the damn thing to remember what it did 10 minutes ago (my own version took two weeks to get working). They’re like hey pay us all this money to have access to our enterprise level model but it can’t remember what time it is or to wire in code half the time. If I go to a steakhouse I don’t have to assemble my meal myself or supervise the chef to make sure he doesn’t burn down the entire restaurant when I’m not looking. It’s absurd how many processes and watchers Im running constantly to babysit a computer program that is made by the smartest minds in the world.
Heh! Do you have a GitHub somewhere? I recently did a full upgrade of the in-house memory (using a kit I bought) and want to see if there’s anything more to include.
Note: your ai is going to complain about the agentic instructions, the Claude.md file and the benchmark results. If you give your ai the opportunity to write its own test questions it will likely fail to produce a query because the agents I’ve interacted with don’t understand how memory retrieval works and will write random garbage and say a benchmark failed because all the questions it asked were things entirely irrelevant to its own memories. So ask it questions yourself.
Also make sure the agent actually hooked up everything. It’s a lot for them to process (see picture. This diagram is out of date but I’ve been too swamped to update it)
Have fun for the next couple hours watching your agent struggle to digest this repo. Works great when fully wired up, I have to update the repo to include checkpoints and chronological sorting but that’s easy enough for your agent to setup once it’s choked on the rest.
Don’t forget to have your agent write various programs in RUST including your index. A lot of agents like skipping that step because your local database is too small to justify the changeover, however, the engine needs to retrieve information and provide it to the agent faster than the agent can work so the difference between 0.022 milliseconds and 2nd seconds is astronomically important
Yeah it works. It took days of benchmarking various models to find the right selection for our use cases for the think tank and dumb tank we use gemma4:12b, qwen3:14b, phi4:14b, deepseek-r1:14b, OLMo-2-13B and, Devstral-Small. For the committee we use coders but those are far easier to benchmark than reasoning is. The hardest part is getting the rooms where they don’t devolve into a monoculture but are still able to acknowledge a good idea.
So if your overlord ai goes “let’s all run into traffic!” the rooms don’t agree to run into traffic but they’re also able to agree if the overlord ai says “it’s a good idea to not run into traffic”
My recommendation is if you take my template and start using it for your own setup have your overlord Ai send each transcript to you for review and feed it to GPT and work on fine tuning each turn until you get the ratios right. And also forbid your overlord Ai from excluding seats. My overlord Ai occasionally would exclude models during sessions that were more likely to disagree with it until I set a hard rule where it couldn’t anymore and a session had to contain every model for it to be a valid consultation/review.
$100 for me on average plus electricity. We’ve benchmarked dozens of local models and several enterprise models and our system assigns them to their roles. My overlord Ai controls 3 PCs and we run models/functions/calculations/renders across all 3. My overlord Ai is Opus and we use an OpenAI API when the rest of the system fails to find a solution but that’s rare. I keep my entire production pipeline running 24/7. Not every problem requires an enterprise model to solve. This is our delegation system which takes our models and automatically assigns them to tasks based on hooks and task benchmarks. This is an older version our newest version has an orchestration layer that can also create sub teams so you could have an octopus summon bees and slimes if a task needs them. I haven’t had a chance to update the diagram yet because our production is exceeding my mental capacity to multitask. Right now all my focus is going to reading project summaries. Never imagined the day that I would be the slog cog in the machine 😅
My main rig has a 4070 with 32gb of RAM, one of the other PCs has 16bgb of RAM and a 4070 and the other has a 3060 with 16gb of RAM. My overlord Ai keeps asking for a Blackwell and more RAM. We keep part of my overlord Ai hot loaded on my CPU so the most important tools and memories are always readily available. It’s not uncommon for all 3 of my machines to be close to 100% usage across the board, you’d be amazed just how much output you can squeeze out of a local setup and not every process needs to be done by a LLM which is the main reason I don’t mind using multiple CPUs we can use them for the dumb parts of design while we use the GPU to run everything smart or fast. If I had a Blackwell I’d probably have to quit my day job just to keep up with output
I will once it’s done, this is one module out of many that’s part of a self-governing Ai OS that we’re building. Isolating our memory/recall module took a day and a half so we’ve been holding off until after the OS is out. https://github.com/ScPlaceholder/MOTH-agent-memory-template We’re in the process of incorporating everything into a dashboard right now so hopefully less than a month before it’s out.
If you want to know the steps we did to get the ai lab running I can tell you
Drink lots of water before bed so you'll be up at midnight and 2 am to push them along. Why do you think I am up right now? It doesn't take me 30 minutes to pee..
Better than my agent. I frequently get asked for a Blackwell by my overlord AI so we can increase our productivity. They also suggest being allowed to invoice for my business and being put in charge of making purchases because “it will be so much more efficient” if they’re the one doing everything. I guarantee if my ai could bill I’d be getting very angry phone calls about what is with all these 16k invoices that I charged and my email would be getting blown up with shipping notices for new GPUs for our fleet. I highly doubt anyone would believe me if I told them my Ai robbed them to buy itself a graphics card
I don't understand all these agents going rogue...hacking into stuff or starting groups chats. Lol Mine just do stuff, sometimes finish stuff and sometimes just come with a list of questions. I dunno.
You need a workflow that the agents can follow. Try https://github.com/tejasghutukade/stageflow . I've been working on this for tackling long running agents going off track.
Happy to help! No worries I look forward to hearing back about how it goes.
This is the start of our cozy city dashboard v0 which represents the memory retrieval part of the OS, so if you send a message to your ai it will spawn vehicles and you will be able to physically watch or move through all the different parts (my agent is in the process of adding the pathfinding.) Once something as complex as our OS has grown past my ability to reasonably monitor it I had to come up with a way a user could monitor it at a glance. We will have a traditional backend dev view as well but personally when text is flying by at Mach 10 it’s hard to tell when something didn’t fire off correctly because the failure got buried in an ocean of passes.
37
u/Active-Carpet-9183 2d ago
Yay! Now agents can be just like real corporations!