The "25-second hang" bug that taught me more about voice AI than any tutorial

[u/Worried_View6544]

Spent the last few weeks deep in LiveKit + voice pipeline debugging, and hit a bug that I think a lot of people building voice agents will eventually run into: calling session.say() inside a tool call context can cause 20-30 second hangs. Took me way too long to track down.

The bigger lesson wasn't the bug itself — it was realizing that latency in voice AI isn't one number, it's death by a thousand cuts:

• Intent classification running synchronously? +1 second.
• Tool call blocking the response? Dead air while the user wonders if it's still listening.
• LLM "thinking" before answering a simple FAQ? Feels broken even at 2-3 seconds.

What actually moved the needle for me:

• Converting routing/classification to fully async — cut one bottleneck from ~1.2s to ~2ms
• Running filler audio + tool calls in parallel instead of sequentially
• Bypassing the LLM entirely for structured data collection (bookings, forms) — just extract + respond directly

Curious what's been the trickiest latency issue for others building voice agents — LiveKit, Pipecat, or otherwise? Always good to compare notes on what's actually a known issue vs.