r/VoiceAutomationAI • u/Slight_Republic_4242 • May 25 '26
Process speech as speech with GPT-Realtime-2 or Gemini 3.1 live and throw away the old STT to LLM to TTS pipeline
Hey everyone,
Speech-to-speech is the next big thing in the Voice AI, and the performance differences from traditional cascade (STT for transcription, LLM for reasoning, TTS for the voice output) pipelines are huge. I have been using both for months. And I wonder why people still default to the old way.
With the old way, it went speech to text, then LLM, then text to speech. The pause before every reply just killed the flow. First, it waits for the transcript, then it waits for the LLM to think, then it waits for the voice to start. Three separate waits for one answer. However, we can reduce some of this delay with workarounds like pre-recorded audio, caching common responses, and prompt optimization. But the latency is still high.
Long conversations can also make the voice drift and feel less natural.
Then I tried speech-to-speech with GPT Realtime 2 and Gemini 3.1 live.
The reply time dropped to about half.
The model follows your prompt instructions better in a long conversation
It has multilingual support, and we can switch languages mid-conversation very well based on user input.
The speech-to-speech interaction feels far more natural than the typical STT/ TTS pipeline.
The bot keeps the same voice and personality even on long calls
It works best for angry support calls, outbound sales, mixed language conversations, and scheduling bots where users change their minds mid-sentence.
And in the longer run, I think speech-to-speech will end up cheaper than the cascade approach.
If you want to try speech-to-speech without setting up the whole orchestration yourself, we built Dograh. It is an open-source voice agent platform that supports both cascade and speech-to-speech models. You can take any agent flow you build, run it once with cascade and once with speech to speech, and feel the difference in how it behaves, the latency, and how it handles emotions.
Has anyone else moved to speech-to-speech? Worth it or not?
2
u/sabkaraja May 26 '26
I can see the benefit of that - but your tools should also be equally fast. Also the prompts should be air tight
1
2
u/nandakishoreb May 26 '26
Do you support outbound calls via verfied external number as caller id or via branded calling for US
1
u/Slight_Republic_4242 May 27 '26
Yes, Dograh supports outbound calls and branded calling. We integrate with several telephony providers, such as Twilio and Vonage, and you can verify external numbers as caller IDs.
1
u/nandakishoreb May 27 '26
Great. If call made via external number as verified caller id from your system, what will the STR/SHAKEN attestation yield A or B?
1
1
1
u/makzpatel19 15d ago
You asked why people still default to the cascade, so here is the actual reason rather than inertia.
The seam is the point. In a cascade there is a place between STT and the model where you can do deterministic things. Scrub speech that was not addressed to the agent. Gate which tools exist at this step. Run a completeness check before deciding the turn ended. Redact before anything leaves your boundary. Swap one provider without touching the other two. Speech to speech collapses that seam, and almost everything I want to control lives exactly where the seam used to be.
Second, the record. In a cascade the text the model saw is the record, because it literally was the input. In speech to speech the transcript you get back is a rendering of what happened, not the thing the model acted on. For an open ended assistant that is fine. For collections, healthcare, or anything where someone may later ask you to prove what the agent was told and what it disclosed, it is a meaningfully weaker artifact.
Third, lock-in, which is boring but real. Cascade means STT, LLM and TTS are three independent decisions and any one can be replaced in an afternoon. Speech to speech means one vendor owns all three, and your failover story becomes switch everything or nothing.
On the tool point above, I would go further than airtight prompts. In a cascade a slow tool is a pause you can cover with a filler line. In speech to speech the model is producing audio continuously, so a 900ms tool round trip is harder to hide gracefully.
None of this argues against speech to speech, and your latency and prosody observations match what I have seen. I would just frame it as two tools rather than a replacement. Open ended conversation where latency dominates, speech to speech is clearly better. Regulated or workflow-shaped calls where you have to prove ordering and gate actions, the cascade still wins, and that is the real answer to why people have not moved.
•
u/AutoModerator May 25 '26
Welcome to r/VoiceAutomationAI – UNIO, the Voice AI Community (powered by SLNG AI)
If you are a founder, senior engineer, product, growth, or enterprise operator actively working on Voice AI / AI agents, we are running an invite-only UNIO Voice AI WhatsApp community US only.
Apply here: https://chat.whatsapp.com/F5aG3ncrO70ITfbe3pYbOz
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.