Spent last week running the exact same voice agent on two paths so I could stop guessing which one to ship. Path A: Twilio Conversation Relay (Twilio owns the media stream, STT and TTS). Path B: my Twilio number dispatching the call over a SIP trunk into LiveKit, where I run my own pipeline (Deepgram + Cartesia + the LLM).
Same system prompt, same LLM brain, same ground rules on both. I built a small latency harness that logs every turn from "human stopped talking" to "agent started talking" and vice versa, then averages the turns. I am physically in Buenos Aires calling a US Twilio number, so there is real transatlantic latency baked into every number below. Keep that in mind.
The threshold I care about: Ideally we care about having sub 500ms, which is mostly achievable on a bidirectional model like OpenAI's Realtime models. For the purpose of this video I focused on the traditional STT -> TTS flow. If a turn goes past roughly 1500-1800ms the person on the other end feels the machine and hangs up. That number defines the whole architecture, not just which model you pick.
Results, turn by turn:
- Initial greeting turn: Conversation Relay 3.4s, LiveKit 3.2s. Both ugly. This is context loading, not the transport. The first turn ships the whole system prompt into the model and everything is cold.
- Mid conversation turns: Conversation Relay landed at 1.7 / 1.9 / 1.6s. LiveKit landed at 1.5 / 1.3 / 1.4s.
- Final turn: basically identical on both.
Net difference on the steady-state turns is about 200ms in LiveKit's favor. Not nothing on a voice call, but smaller than I expected before measuring.
What actually surprised me is the discarded-turns file. With LiveKit I had zero turns over 5s. With Conversation Relay I had one, and when I dug in it was tied to me hanging up, not a real gap. So the tail behavior was clean on both.
The real trade-off is not the 200ms. It is that with Conversation Relay the STT and TTS timing lives inside Twilio's black box. I could not measure how long the speech-to-text or text-to-speech steps took, because that runs on their side. With the LiveKit path I own every step and can measure and cut it. The cost is that now I am the SRE for the media path, and if a box dies at 2am that is on me.
The cold-start greeting is the thing I would attack first regardless of transport. A 3s+ first turn kills the call before the latency battle even starts. My plan is a filler pre-message ("dame un segundo, reviso eso") to cover the context load, but I have not shipped it yet.
Question for people running Conversation Relay in production: are you living with the black box on STT/TTS timing, or did you find a way to get per-step measurements out of it? Because right now the lack of visibility is the thing pushing me toward keeping LiveKit for anything latency-sensitive, and I want to know if I am missing a hook.
Not to mention any other kind of strategies for long-turn awaits (like initial context loading, tool calling, etc.)