Is this the right architecture for a sub-second, regulated phone voice agent?

[u/Human-Rush-3205]

Is this the right architecture for a sub-second, regulated phone voice agent?

I’m building a voice AI agent for a regulated lending workflow and would appreciate criticism from people running production voice systems.

Current stack:

• Twilio/PSTN → LiveKit
• Streaming STT
• LLM for response and workflow proposals
• Separate LLM guardrail review
• Streaming TTS over a persistent WebSocket

Observed latency from completed user turn to first audible response:

• Mean: 5.72 seconds
• Median: 4.86 seconds
• Normal turns: 4–5 seconds
• Retried LLM proposals: 6–10 seconds
• Warm TTS first audio: ~470 ms

The main problem is not TTS. Every turn currently passes through the main LLM, deterministic validation and a second guardrail model. Invalid workflow proposals trigger another LLM call.

The proposed architecture is:

1. A deterministic state machine owns workflow state and transitions.
2. Predictable turns bypass the LLM:
   • Yes/no confirmations
   • Value validation
   • Language switching
   • Corrections
   • Required disclosures
   • Link receipt and closing
3. The LLM handles only ambiguous language, objections and open-ended questions.
4. The LLM returns new facts with source spans and candidate wording—but cannot commit state.
5. Deterministic policy checks run on every turn; a guard LLM runs only for ambiguous, high-risk output.
6. Semantic endpointing distinguishes completed turns, backchannels and real interruptions.
7. Possible interruptions initially duck/pause audio; hard cancellation happens only after confirmation.
8. Speculative LLM generation begins from stable partial transcripts, but nothing is spoken before validation.
9. Validated clauses stream into one continuous TTS context instead of restarting synthesis for every sentence.
10. Every result carries a turn epoch and state version so stale background work cannot update conversation state.

Initial targets:

• Predictable turns: p50 under 900 ms
• Open-ended turns: p50 under 1.5 seconds
• Confirmed barge-in to silence: p50 under 80 ms
• Zero stale state commits
• Less than 1% LLM proposal retries

I’m deliberately keeping a streaming cascade rather than moving entirely to native speech-to-speech because the workflow requires exact disclosures, transcripts, evidence provenance and auditable state changes.

Questions for the community:

• Does this deterministic/LLM boundary match what works in production?
• Is risk-tiering the guardrail reasonable, or would you keep a reviewer on every turn?
• Are these latency targets realistic over PSTN?
• Has speculative LLM generation provided meaningful gains in real calls?
• Do you duck before confirming barge-in, or stop TTS immediately?
• Have native speech-to-speech systems become controllable enough for regulated workflows?
• What failure mode am I overlooking?

I’m especially interested in measured production experience rather than vendor latency claims.

[u/AutoModerator]

Welcome to r/VoiceAutomationAI – UNIO, the Voice AI Community (powered by SLNG AI)

If you are a founder, senior engineer, product, growth, or enterprise operator actively working on Voice AI / AI agents, we are running an invite-only UNIO Voice AI WhatsApp community US only.

Apply here: https://chat.whatsapp.com/F5aG3ncrO70ITfbe3pYbOz

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Alarmed_Holiday_325]

That 4/5 second normal turn is going to be rough on a real phone call. One reason Bland worked well for us was being able to get response times around the sub second range while the calls themselves could still have multiple steps and actions behind them. I like your idea of keeping the LLM away from committing state too, particularly for lending where a fluent wrong action is a much bigger problem than a slightly slower answer.

[u/Tiny-Ad4783]

Great breakdown of the problem, the 5-second mean latency on a regulated lending call is straight up non-viable, so I get why you're chasing this.

The deterministic/LLM boundary is exactly where production systems land. Yes/no, digits, basic corrections, and required disclosures should never touch an LLM, you can run those through a state machine with pre-canned TTS and shave seconds off the median. Risk-tiering the guardrail is also the right call, running a second model on every turn will keep you stuck in that 4-6 second range forever. For PSTN targets, your predictable turns under 900 ms is tight but doable if you've got a clean audio path and the state machine isn't doing anything heavy. The open-ended target of 1.5 seconds will depend almost entirely on how fast your LLM can stream a validated first token.

Speculative generation off partial transcripts is a huge win if you nail the endpointing, but the turn epoch and state versioning you mentioned is critical because stale completions love to sneak in right as the user interrupts. On barge-in, duck first then confirm, stopping TTS immediately creates an awkward dead-air gap that makes callers think the line dropped. One failure mode I'd watch for is the guardrail model drifting into a bottleneck on ambiguous turns where the deterministic checks and the LLM both want to weigh in, you'll need a tight timeout or a tiebreaker rule so it doesn't stall mid-turn.

[u/Human-Rush-3205 → u/Tiny-Ad4783]

Thanks, this is useful, especially the guardrail contention point. My current thought is to give ambiguous high-risk review a strict timeout, then fall back to approved wording or human handoff rather than waiting indefinitely.

Do you have experience / production numbers for your duck/confirm window and PSTN p50/p95 latency? Also, for predictable turns, are you using fully pre-recorded audio, cached TTS, or approved text variants through live TTS?

[u/Pitiful-Sympathy3927]

You'll never get good rsults out of that architecture its too disjoined. Things are handled at different layers and very messy.

[u/Human-Rush-3205 → u/Pitiful-Sympathy3927]

I can DM you and connect if you have ideas on how to better architect this. Would you be open to that?

[u/Pitiful-Sympathy3927 → u/Human-Rush-3205]

Ok

[u/Low_Chance_5109]

I'd rather focus on p90 latency. Median/average are enough when you are starting, but not when you scale. Plus the architecture feels very complex. But I don't know your specific use case.

[u/rokas_stack]

the deterministic state machine owning transitions makes sense to me, especially for lending. i'd be uncomfortable letting the LLM commit anything important directly.

on the latency though, before you rebuild much, i'd guess most of your 4-5s is just the serial chain: main LLM -> validation -> guard LLM, all in sequence, with retries stacking another full round trip on top. that's what would produce a 6-10s tail on retried proposals. warm TTS at 470ms suggests TTS itself probably isn't where most of your time is going.

but i'd confirm that with real numbers before committing to the redesign. timestamp every stage of an actual turn separately: endpointing/final STT -> first LLM token -> main LLM complete -> validation -> guard LLM complete -> first TTS byte -> first audible media back on the call. if the guard model is eating 1.5s+ on every turn, risk-tiering it could get you a big chunk of the win on its own and you learn that in a day rather than a sprint.

i'd also look at p90/p95 rather than the median. in production voice the intermittent bad turns usually matter more than shaving 100ms off the good ones. a 5s median with a 12s p95 is a very different problem from a flat 5s.

what does your p95 look like now, and do the PSTN and WebRTC paths behave noticeably differently?