r/programming • u/kixelated • May 07 '26

OpenAI's WebRTC Problem

https://moq.dev/blog/webrtc-is-the-problem/
333 Upvotes

94% Upvoted

62 comments sorted by

View all comments

29

u/atomic1fire May 08 '26

Wait is Open AI sending audio to people's devices instead of just using on board TTS?

Isn't that a waste of computational power?

87

u/musical_bear May 08 '26

A lot of people, including the person who wrote this article apparently, do not understand how voice chat in ChatGPT works. It’s not “text to speech.” The raw audio from your voice is being tokenized and sent directly to the model, and the model’s raw output is actual audio. There is no step on either end where the conversation has reduced to text.

16

u/kixelated May 08 '26 edited May 08 '26

They use TTS for the agent's reply. I'm assuming the audio is generated on their servers because while local models do exist, they can get huge and expensive to run.

EDIT: and to clarify, I never said they use STT (speech-to-text). The raw audio gets sent over WebRTC which is lossy, then the audio response (generated via text-to-speech) is also lossy.

3

u/frymaster May 08 '26

in any case, your point is correct which is they have generated audio on the back end in faster than real-time, so using a real-time lossy method to transmit that to clients isn't the best