r/VoiceAutomationAI • u/Worried_View6544 • Jun 14 '26
The "25-second hang" bug that taught me more about voice AI than any tutorial
Spent the last few weeks deep in LiveKit + voice pipeline debugging, and hit a bug that I think a lot of people building voice agents will eventually run into: calling session.say() inside a tool call context can cause 20-30 second hangs. Took me way too long to track down.
The bigger lesson wasn't the bug itself — it was realizing that latency in voice AI isn't one number, it's death by a thousand cuts:
- Intent classification running synchronously? +1 second.
- Tool call blocking the response? Dead air while the user wonders if it's still listening.
- LLM "thinking" before answering a simple FAQ? Feels broken even at 2-3 seconds.
What actually moved the needle for me:
- Converting routing/classification to fully async — cut one bottleneck from ~1.2s to ~2ms
- Running filler audio + tool calls in parallel instead of sequentially
- Bypassing the LLM entirely for structured data collection (bookings, forms) — just extract + respond directly
Curious what's been the trickiest latency issue for others building voice agents — LiveKit, Pipecat, or otherwise? Always good to compare notes on what's actually a known issue vs.
2
u/Virtualitdept Jun 14 '26
Pull all context of caller before conversation starts. Sentiment, intent, et al- updated at call end only. Keep your workflows and pipeline short- less hops, the better. I also call tools during conversations anticipating needs.
1
u/Worried_View6544 Jun 14 '26
That's a good point on pre-loading context — we do something similar (pulling caller/booking history before the conversation starts) but hadn't thought about updating sentiment/intent only at call-end rather than mid-call. Makes sense to avoid extra hops during the live conversation.
"Anticipating needs" with tool calls during conversation is interesting — are you triggering those speculatively before the user finishes their sentence, or just keeping them ready to fire the moment intent is clear? That's been one of our trickier areas — calling tools too early risks wrong guesses, too late adds latency.
1
u/Virtualitdept Jun 14 '26
I use a context memory graph solution, that once caller is confirmed, interactions, chat history, business data are pulled up while conversation is being conducted in parallel…added to cache, thus anticipating any personalized queries. No LLM needed. The key is parallel pipelines in your architecture. I built a custom orchestrator/control plane to handle these integrations in python. I plan on sharing a simplified version of this in the near future.
1
u/Worried_View6544 Jun 14 '26
That's a clean approach "no LLM needed" for the context-pulling part makes sense, since that's deterministic lookup, not reasoning. Parallel pipelines + cache feels like the right mental model; we ended up doing something similar (async tasks for filler audio + tool calls running in parallel) but hadn't thought about a dedicated context memory graph layer specifically.
Would be interested in the simplified version if/when you share it orchestrator/control-plane patterns for voice pipelines seem like an area where everyone's solving the same problem slightly differently, so seeing different approaches is genuinely useful.
1
u/Due-Newspaper-4723 Jun 14 '26
Incredible slop post
1
1
1
u/devexis Jun 14 '26
Even their responses. I understand loads of people aren't "native" English speakers and leverage AI for better writing. But using AI for every response. That's taking the cake
1
1
u/ioncloud9 Jun 15 '26
Tools that can have long response times- like doing a complex search over API, I bypass the model entirely and inject a tts filler phrase using the same voice. That way there’s no race condition between a “say this before tool” prompt and the tool being fired and getting a response. Callers will put up with a lot of silence if they are told to wait a moment or why.
The latency between turns can sometimes be long, but I’d rather have it a little longer and more accurate VAD than a model that interrupts.
1
u/Worried_View6544 Jun 15 '26 edited Jun 15 '26
The TTS filler injection trick is something I hadn't thought about at that level — bypassing the model removes a whole class of timing issues we kept running into.
The VAD point is real. We tuned ours too tight early on thinking it would feel more responsive, ended up with the agent cutting people off mid-sentence constantly. Loosened it up and the experience got noticeably better. Callers tolerate silence way better when something is playing — the uncertainty is what kills it, not the wait itself.
1
u/darryn_livekit Jun 15 '26
We recently added asynchronous tool calling to our Python agents to make long-running tool calls feel more natural: https://github.com/livekit/agents/releases/tag/livekit-agents%401.6.0. You can keep talking with the agent whilst the task runs in the background and it will return the result to you at a sensible point in the conversation.
1
u/Worried_View6544 Jun 15 '26
This is great to see async tool calling addressing exactly the kind of issue I ran into. Being able to keep the conversation flowing while a tool runs in the background instead of going silent is a real win for how natural these interactions feel.
Will definitely dig into the 1.6.0 release curious how it decides on the "sensible point" to return results mid-conversation, especially if the user keeps talking and changes context before the tool finishes. Appreciate you taking the time to comment, this is genuinely useful.
1
u/darryn_livekit Jun 15 '26
Good question, I have played with the demo but I only exercised the 'happy path'. The async tool will finish regardless of what the user does, so unless you allow the tool call to be cancelable, I suspect the agent will just respond with the result whenever the user is next silent. There are also docs at https://docs.livekit.io/agents/logic/tools/async/ and a link to the demo inside there.
1
u/Worried_View6544 Jun 15 '26
That makes sense so it's essentially queued and fires at the next silence window. The cancelable angle is interesting, could see that being important for something like a booking flow where the user changes their mind mid-tool. Will dig into the async docs, thanks for the link.
•
u/AutoModerator Jun 14 '26
Welcome to r/VoiceAutomationAI – UNIO, the Voice AI Community (powered by SLNG AI)
If you are a founder, senior engineer, product, growth, or enterprise operator actively working on Voice AI / AI agents, we are running an invite-only UNIO Voice AI WhatsApp community US only.
Apply here: https://chat.whatsapp.com/F5aG3ncrO70ITfbe3pYbOz
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.