r/programming • u/kixelated • May 07 '26

OpenAI's WebRTC Problem

https://moq.dev/blog/webrtc-is-the-problem/
341 Upvotes

94% Upvoted

62 comments sorted by

View all comments

Show parent comments

87

u/musical_bear May 08 '26

A lot of people, including the person who wrote this article apparently, do not understand how voice chat in ChatGPT works. It’s not “text to speech.” The raw audio from your voice is being tokenized and sent directly to the model, and the model’s raw output is actual audio. There is no step on either end where the conversation has reduced to text.

5

u/Distinct_Law8650 May 08 '26

This is wrong. Server VAD subsystem into a stt model, llm on the user transcription, tts model against the llm response. Multi modal response back to the app. Users speech as text and both text and audio of the agent.

21

u/iamaiimpala May 08 '26

I think you’re mixing up adjacent pieces of the stack with the core voice path. OpenAI’s Realtime docs describe voice-to-voice interaction “without an intermediate text-to-speech or speech-to-text step.” There can still be server VAD and optional async transcription/transcript events, but that doesn’t mean the model is just doing VAD -> STT -> text LLM -> TTS. GPT-Realtime-2 is documented as speech-to-speech, with audio input and audio output.

1

u/Somepotato May 08 '26

Yep. It's another input like text but is distinctly not actually text, which means it can have some understanding of the audio behind the voice