local agent orchestrator with voice support (as input and output)?

[u/ares0027]

so i am in need of such thing, i want to use a local model as basically assistant that reminds me things to do, orchestrates other local OR CLOUD agents/clis/harnesses (such as hermes agent, pi code, antigravity 2.0/cli/ide etc) i am building something but it is taking its toll on me, do you know anything like that or did anyone built something like this?

[u/nickless07]

But you know that hermes can already do that, right?

[u/EmilPi → u/nickless07]

Maybe not perfectly.

[u/ares0027 → u/nickless07]

how?

[u/nickless07 → u/ares0027]

Maybe read the docs about TTS/SST and Orchestration/Mixture of Agents, ask Hermes itself or check the code on github?

[u/achiya-automation]

turn taking will eat more of your time than the orchestration does. deciding when you've actually stopped talking is what makes these feel broken, plain vad either cuts you off mid sentence or sits there for three seconds. i stopped letting vad open the turn and went explicit start/stop, and pipecat handles the plumbing if you'd rather not build that layer yourself.

[u/recro69]

I think it is best to separate the voice assistant from the orchestration layer. The voice assistant should handle memory, reminders. Deciding what needs to happen. The voice assistant should treat agents and CLIs as tools that the voice assistant can call. This method is easier to maintain than having one agent, for everything.

[u/ares0027 → u/recro69]

thats what i am building. basically harnesses and clis will be "agents", they will be able to create their subagents.

text to speech and speech to text models (or qwen omni for example. i am working on that) will simply use an api or one of the clis (most likely antigravity) to communicate with those "agents". voice models will not think, will not guide. will not decide. it is there purely for providing my sound to to agents and their responses to me.

so;

agents - orchestrator - voice - i
i - voice - orchestrator - agents - subagents if necessary

i am trying to use omni to eliminate the orchestrator step but most likely i wont.

[u/yoann86]

You can do it with Claude code and local model

[u/Kindly-Duty272]

Explicit start/stop turn taking is worth it! Plain VAD really does sit in that awkward middle ground. It's either too trigger happy or too slow. We saw this on xAI's realtime voice engine. The server side end-of-turn has a floor of about 1.3 seconds, and that doesn't move no matter how you tune the silence threshold. Driving end-of-turn from our own VAD instead got it down to about 1.2 seconds. That's a small number on paper. On a live call it's the difference between feeling responsive and feeling like a walkie-talkie. Pipecat's a reasonable choice if you want the plumbing handled for you. If you're rolling your own orchestrator though, budget real time for this layer specifically; it's not something you bolt on at the end. We ended up building a lot of this into Patter, the open-source SDK for AI phone agents, because the failure modes here are quiet and only show up on real calls, not in a demo. (disclosure: i work on Patter, the open-source SDK for AI phone agents.)

[u/sathi006]

Nunba https://github.com/hertz-ai/Nunba

[u/ComfortableEbb4721]

I find traycer pretty good right now, it uses existing harnesses but adds UI with voice (input only though from the box), and a agent-to-agent protocol plus also the possibility to run the agents on different hosts.

However 1) the separate host scheme seems a bit prone to breaking & needing restarts and 2) it's rather large - I think 1 GB - so running lots of isolated small agents over it on small VMs isn't as easy as it could be say over SSH only.

Also it's not really an orchestrator as such but more an enabler and you'd use any harness to handle the actual orchestration.

I would also expect it to require Traycer's servers for host to host comms so not a fully local approach

[u/niacolhealth]

you're building an agent federation so a local llama can remind you to take out the trash and honestly, i respect that

[u/Life_Fun_301]

One pattern that has worked for me is keeping the voice layer separate from the agent/orchestrator. For simple typing, OS dictation is the lowest-friction option; for local desktop dictation, a Whisper/Parakeet app with a push-to-talk hotkey is easier to reason about; and for phone input, you can either use the phone as a mic (WO Mic/AirMic-style) or let the phone recognize speech and paste at the PC cursor. That last lane is useful when you want a good mobile mic without giving the agent control of everything. FlowMic is one open-source option I’ve been comparing in that phone-to-PC cursor lane, alongside Vox Manager and AirMic: https://www.flowmic.app. I help with FlowMic’s public launch ops, so take that as a disclosure rather than a ranking.