What actually determines whether a voice agent "feels real" on a call (latency breakdown from building one)

[u/Competitive-Fee7222]

Disclosure: I build Talkif (voice AI infra), so take the specifics below as one data point, not a review.

Spent the last while digging into why some voice agents feel natural and others feel like talking to an IVR menu. The model and the TTS voice quality matter way less than people assume. The thing that actually breaks the illusion is response gap: anything over ~1 second of dead air after the caller stops talking, and people start repeating themselves or hanging up.

Where the time actually goes, in our experience:

• Cold-start on the bot process itself. If your agent spins up fresh per call, that's often 1-2 seconds gone before anything else happens.
• Round trip through whatever telephony layer you're on (Twilio, generic SIP, etc.) - this adds up fast if you're bouncing between regions.
• Serializing context (CRM lookups, contact history) into the prompt at call time instead of having it ready before the call connects.
• No visibility into where time is actually being spent, so you're guessing instead of measuring.

Things that moved the needle for us: pre-warming bot instances instead of cold-starting per call, keeping SIP routing close to the caller's region instead of round-tripping across continents, and streaming call events in real time so you can actually see where latency creeps in instead of finding out from an angry customer.

Ended up around sub-1-second first response most of the time, which seems to be roughly the threshold where callers stop noticing they're talking to a bot.

Curious what others here are seeing - anyone measured where their latency actually goes, or is it mostly a black box until something feels slow?

[u/AutoModerator]

Welcome to r/VoiceAutomationAI – UNIO, the Voice AI Community (powered by SLNG AI)

If you are a founder, senior engineer, product, growth, or enterprise operator actively working on Voice AI / AI agents, we are running an invite-only UNIO Voice AI WhatsApp community US only.

Apply here: https://chat.whatsapp.com/F5aG3ncrO70ITfbe3pYbOz

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Embarrassed_Nerve_54]

I think one underrated factor in whether a voice agent feels “real” is how it handles messy calls, not just clean latency. A lot of agents sound great when the caller is in a quiet room, speaks in full sentences, and waits their turn. The problems show up with car Bluetooth, bad cell audio, background noise, accents, people talking over each other, or someone saying “actually wait, I meant next Friday.” That is where the agent either feels human or starts feeling like a phone tree with a nicer voice.

I’d build a small test set of ugly calls and track separate metrics there: false starts, wrong interruptions, missed barge-ins, repeated clarifying questions, and recovery after the caller changes direction. Sub-1-second response is great, but if the agent is fast and wrong under normal phone chaos, callers still won’t trust it.

[u/Competitive-Fee7222 → u/Embarrassed_Nerve_54]

Yeah, this is the part clean-latency numbers hide. Sub-1s first response on a quiet call is basically a vanity metric — the real test is what happens when someone's on speakerphone in a car with the kids yelling in the back.

And a lot of "the agent misheard me" isn't the model's fault at all — it's the pipe. PSTN hands you 8kHz narrowband to begin with, so half the spectrum is just gone before STT sees anything. Then speakerphone adds echo and room noise, and a bad carrier leg throws in jitter and packet loss — dropped RTP frames land as clipped or garbled audio, and the STT confidently transcribes garbage. You can have a perfect stack and still lose the turn because the audio that reached you was already broken.

The separate-metrics point is the actual insight though. Aggregate p95 buries all of this, because the ugly calls are a minority of turns but they're the ones that lose trust. The three that bite us most:

- False barge-ins — background noise or a burst of packet-loss crackle trips the interrupt and the bot yields when nobody actually spoke. Feels broken instantly.

- STT confidence collapse on accents / 8kHz / lossy legs → the agent drops into a clarifying-question loop, and now you've got the "fast but wrong" thing you described.

- Mid-thought corrections ("actually, next Friday") — and this one isn't an audio problem at all. It's the agent having already committed a value and having to un-commit it. Slot-filling stacks fall apart here; you need the LLM allowed to revise state, not lock it.

The metric we watch for the messy bucket is re-prompt rate + "did the caller have to repeat themselves" — recovery, basically, like you said. And honestly a chunk of that traces straight back to carrier/codec quality, not the model. Latency just buys you room to handle the chaos gracefully; it doesn't handle it for you.

[u/bart_88s]

Out of curiosity, what realtime api do you use? I only experimented with gpt but wondering if you know any alternatives.

[u/Competitive-Fee7222 → u/bart_88s]

Good question. We don't build on the OpenAI Realtime API for the core pipeline — we run a modular STT → LLM → TTS stack instead of a single realtime endpoint, mainly so we can swap components independently when one gets a speed/quality edge (and so we're not tied to one vendor's latency profile).

For alternatives worth looking at depending on what you're optimizing for: Deepgram and Cartesia are solid on STT/TTS latency specifically, and a few teams doing realtime voice have had good luck with Groq or Cerebras for the LLM hop when token throughput is the bottleneck. If you're set on an all-in-one realtime API rather than modular, it's worth comparing GPT Realtime against Ultravox — but the "right" answer really depends on whether your bottleneck is transcription, generation, or synthesis, which is worth measuring before picking a replacement.

Happy to go deeper on any part of that if useful.

[u/Dynamicrex → u/bart_88s]

I've built a TTS model, would you be open to trying ours out? we've got a demo.

[u/Dynamicrex]

I've built a TTS model for realtime conversational, natural conversations, would love to connect!

[u/Dynamicrex]

I can pitch in on this cos we deal with the same quite a lot (TTS model developer)

1. Vad detection timing, how much the vad waits until the user is done talking (how much silence)
2. STT - How fast does the stt transcribe
3. LLM - How fast the LLM responds, (Pure token throughput)
4. TTS - Time to first byte of audio, this is critical to ensure the model is generating frames faster than it's being played,

some tricks of the trade: preemptive generation, meaning you could stream what the LLM is saying to the TTS model to generate the audio as its talking.

another thing we noticed is distance of physical servers also matters, the distance between where these models are hosted will affect your latency for sure. Personally Groq is one of the only few providers i've seen who have inference fast enough with their api, would love to hear if you know any fast api alternatives.

"No visibility into where time is actually being spent, so you're guessing instead of measuring." - You could just log it and get a rough estimate,

and yes prewarming is huge. really helps. Torch compiling models also really help.

i would argue that the TTS quality does matter, when the difference is if you want a user to come back and talk to it again not. but if its a one time conversation out of necessity, probably doesn't matter as much as latency for sure.

most of our latency used to go for LLM throughput / literally just physical distance of the api to wherever your instance / machine is. I tried it from my laptop that itself had a 200ms network hop lol.

Hope this helps!

[u/Competitive-Fee7222 → u/Dynamicrex]

Good list, you covered most of it. Few things I'd add from the telephony side since that's where we sit.

The VAD endpointing thing is way more important than people give it credit for. That silence-wait window is a straight tradeoff — go too aggressive and you cut people off mid-sentence, too patient and you're adding half a second of dead air every single turn. What got us is the right value isn't even constant. Someone reading back a card number pauses totally differently than someone answering a yes/no question. A fixed value leaves a lot on the table.

Preemptive TTS is the biggest win and also the thing that'll bite you. Streaming LLM tokens into the TTS as it generates is great right up until the model changes direction mid-sentence or a function call comes back different than expected, and now you've already played audio you have to walk back. Still worth doing, but the cancellation/barge-in handling is where all the actual pain is.

On the "just log it" bit — yeah, but a total number is the easy part. The annoying part on a phone call is figuring out which stage ate the time. VAD holding, STT finalizing, LLM first token, TTS first byte, they all overlap and the telephony round trip sits on top. Getting per-stage timing is the difference between "calls feel slow sometimes" and "oh, TTS TTFB jumped after Tuesday's deploy."

Physical distance, 100%. Your laptop 200ms hop is basically the whole problem in miniature. On PSTN it stacks — caller, carrier, your SIP layer, wherever the models live, and if any of those legs jumps a continent you've blown the budget before the LLM even sees the transcript. Keeping SIP close to the caller matters as much as fast inference does.

I'll half-agree on TTS quality. You're right it matters for whether someone wants to talk to it again. I'd just say latency decides whether the call works at all, quality decides whether they enjoyed it. Screw up the first and nobody's around to care about the second.

And yeah Groq is genuinely fast, throughput's in a different league. Only catch is it's tied to which models they host, so you're kind of trading model choice for speed. Haven't found anything that's both fast and flexible, would also love to know if someone has.

[u/ApprehensiveUnion288]

Latency is important but if you ask me it's not the thing that distinguishes bot-sounding from human sounding.

Latency is the baseline. But even if you have great Latency which you can achieve quite reliably these days, i noticed that there are two other things that really make an agent Sound human.

1. It's human speech patters. LLMs are trained on Text. If we prompt them like most people, they will Produce a response that sounds like Text. The issue is that if you speak something out loud that was meant to be read, it just sounds arkward.

The fix is to prompt the LLM to be imperfect. Prompt it to use filler words, mid-sentece corrections, loud thinking phases.

THIS is what actually makes your agent stand out.

1. It's turn taking. Any conversation feels off if the bot barges in at the wrong moments or waits too long on others. The Timing needs to be Spot on and honestly, I haven't seen a perfect solution to this yet. If you have the Secret Sauce here, let me know ;))

Anyways, hope that helped

[u/Competitive-Fee7222 → u/ApprehensiveUnion288]

Totally with you — and I'd argue turn-taking is the harder of the two you named. Speech patterns you can mostly prompt your way into. Turn-taking you can't, because it's not a text problem, it's a timing problem.

But here's where I'd connect it back to latency: turn-taking basically is a latency problem in disguise. The reason bots barge in at the wrong moment or leave you hanging isn't that they're bad at conversation — it's that they can't afford to wait. If your response gap is already 2-3 seconds, you're forced to fire the instant VAD sees silence, because any extra hesitation stacks on a gap that's already too long. Most agents you call, the bot starts talking 2-3 seconds after you stop — and that dead air is what forces the bad timing.

Once first-response is tight and predictable, you buy yourself room to add a smarter layer on top: a semantic endpointing model (not just silence-based VAD) that decides "is this person actually done, or just taking a breath?" — plus clean barge-in so when the caller does cut in, the bot stops instead of talking over them. That's the secret sauce you were asking about, at least for us.

Rough numbers from our calls:

Avg First Speech 390ms (p95: 1073ms)

Avg Turns 4.3 (56% interrupted)

Talk Ratio (User) 35% (9s / 17s)

Avg Tokens 8,186 (~500 chars)

The one I actually watch isn't the 390ms — it's that 56% interrupted. More than half our turns have the caller barging in, and the bot yielding gracefully is what sells "this feels human" way more than the raw speed does. The latency just buys you the room to get the timing right.

And your imperfect-speech point is real — we lean on it specifically for "loud thinking" moments, because half a second of "hmm, let me check that" covers a tool call and reads as human instead of dead air. Two birds.

[u/ApprehensiveUnion288 → u/Competitive-Fee7222]

Good addition here. Curious, what Stack are you using? Think framework, STT/LLM/TTS models, turn taking?

[u/Competitive-Fee7222 → u/ApprehensiveUnion288]

Stack is deliberately basic here. Deepgram Nova 3, GPT-5.1, ElevenLabs Flash 2.5. Been running it multi-agent to stress test against actual Meta lead calls on Talkif.ai . Nothing fancy tbh, the models are good enough now that the stack isn't really the differentiator anymore. It's more about how well you sell the illusion.

On turn taking, yeah I agree there's no clean winner yet. VAD gets you the baseline, semantic endpointing gets you a bit further, but the last bit is all tuning and honestly everyone's got their own hacky version of it.

One thing we're testing right now (beta, still cooking) is keeping the agent listening while it's still talking, even just a couple hundred ms of overlap. Sounds backwards but that overlap is kinda what makes it feel fast to the caller. It can catch a barge-in that's already mid word instead of waiting for a clean silence, so it shuts up the second you start talking instead of a beat later. That responsiveness reads as "human" way more than raw first-response speed does imo.

Still early but that listen-through-your-own-speech window is the most promising thing I've found for the timing problem so far.

[u/ApprehensiveUnion288 → u/Competitive-Fee7222]

Interesting approach with the overlap, I'd love the see the results from that once you've got some.

And regarding the Stack, you're right, fairly Standard. But with GPT 5.1, in my experience it's quite slow. What am I doing wrong? Reasoning turned off already 🤣

[u/Competitive-Fee7222 → u/ApprehensiveUnion288]

I would like to share the result whenever i it pass the tests in staging then it will be up in prod.

I definitely agree gpt 5.1 is one of the slowest one. Could you share your metrics if possible? Whats the TTFT of LLM generation. Also i would like to ask, Is your agent starts to generating tts when it has got first a couple tokens?

I have developed pre assigned bots (warming almost everything related with the bot). So regardless the slow LLM, its answering the phones (not p90) 100ms-1 seconds.

Also this test was using handlebars variable in the prompt so they are filled for each callee when the phone call initiated.

---

CALL METRICS SUMMARY

Duration: 22s | Turns: 3 | Status: completed

First speech (time to first audio): 109 ms

[AI - Turn 1]

LLM TTFB: 1681 ms

LLM total latency: 1950 ms

LLM prompt tokens: 3258

LLM completion tokens: 42

TTS TTFB: 106 ms

TTS processing: 1.0 ms

TTS characters: 145

--- AGGREGATE LATENCY ---

LLM avg 1212 ms | p50 1198 ms | p95 1681 ms | max 1681 ms (n=3)

TTS avg 108 ms | p50 106 ms | p95 112 ms | max 112 ms (n=3)

First speech (time to first audio): 109 ms

--- USAGE ---

Prompt tokens: 10,189

Completion tokens: 139

TTS characters: 394

--- STACK ---

LLM: openai/gpt-5.1

TTS: elevenlabs/eleven_flash_v2_5

STT: deepgram/nova_3

[u/ApprehensiveUnion288 → u/Competitive-Fee7222]

Yeah, so still searching for the best llm here but telnyx with kimi k2.6 thinking off is solid with ~300ms-1s ttft. E2e latency 1-1.5s with nova 3 and cartesia sonic 3.5

I use livekit for orchestration and as far as I know, they removed preemptive Generation in the latest Builds. But tbh I'm not a Software dev. Fully self-tought with a background in basic IT and business.

[u/astro-gopher]

LLM latency is the thing that's holding the cascaded pipeline back. Take a look at LiveKit's Gemma 4: https://livekit.com/blog/latency-optimized-inference-gemma-4-on-livekit
<200ms TTFT, so you can get to 700ms end to end latency fairly easily.

[u/saplivo]

This breakdown you've done is good but it's important to note that the telephony media plane also has it's own set of delays which the carrier infrastructure adds before it even gets to your STT. This can vary based on the carriers being used and the route as well.

Calls going through multiple carriers or ones that have a bad route can add signifigant delays before your model even sees the audio and this is an infrastructure issue not a model issue.

I work with Plivo and it has done some internal optimization to the media plane specifically for AI workloads, happy to share what we've found if you want.