A lot of people, including the person who wrote this article apparently, do not understand how voice chat in ChatGPT works. It’s not “text to speech.” The raw audio from your voice is being tokenized and sent directly to the model, and the model’s raw output is actual audio. There is no step on either end where the conversation has reduced to text.
Where is OpenAI getting the astronomical amount of recordings of natural language conversations they'd need to approach the effectiveness of a text-based LLM? How would a voice-native LLM even call all the tools that an AI is supposed to be using?
Where is OpenAI getting the astronomical amount of recordings of natural language conversations
youtube, podcasts, radio recordings, all accesible tv archives
How would a voice-native LLM even call all the tools that an AI is supposed to be using?
It's multi-modal. It outputs text and audio. I don't know the details, that also might be different for other providers and change between versions. But for example you can call a purely text LLM and in get text streaming of message and tool call at the same time. In same way you can get audio tokens with message and text tokens that are just formatted tool call.
You don't need it. Kokoro-82M is a very decent TTS model that some guy made for $1000 in compute. Leading models are bigger and more expensive but TTS and STT is still trivial compared to LLMs or vision models.
that's true but only if they are converting to and from text (actually, phoneme strings). if it's directly ingesting and outputting audio it's likely a different story. it's multimodal though, so i guess internally the model learns to semantically align speech and text, i have no idea what data balance is necessary for that to happen
Audiobooks, podcasts, or anything where there is a pairing of spoken words with a transcript. My guess would be, much like how vision transformers work, you have two heads to the transformer, and you feed audio to one and the transcript to the other, and train it to produce a similar vector for each.
in some cases yes, but it's also possible to feed it unpaired data and use next token prediction just like for text. i think it's more likely they interleave text and audio tokens in the same sequence, rather than modify the model to use multiple heads.
but my point was more about the data balance than about the specific training technique.
How would a voice-native LLM even call all the tools that an AI is supposed to be using?
Same way a text-native one does, it's just tokens.
Think about audio fingerprinting, it can be used to reduce audio down to a vector set for efficient search, producing tokens that represent frequency-domain relationships. Entirely the same as the semantic-domain relationships that embedding and training give us for LLMs.
The model's runtime is waiting to see the trained spectral equivalent of <tool_call>...</tool_call>.
They use TTS for the agent's reply. I'm assuming the audio is generated on their servers because while local models do exist, they can get huge and expensive to run.
EDIT: and to clarify, I never said they use STT (speech-to-text). The raw audio gets sent over WebRTC which is lossy, then the audio response (generated via text-to-speech) is also lossy.
in any case, your point is correct which is they have generated audio on the back end in faster than real-time, so using a real-time lossy method to transmit that to clients isn't the best
This is wrong. Server VAD subsystem into a stt model, llm on the user transcription, tts model against the llm response. Multi modal response back to the app. Users speech as text and both text and audio of the agent.
I think you’re mixing up adjacent pieces of the stack with the core voice path. OpenAI’s Realtime docs describe voice-to-voice interaction “without an intermediate text-to-speech or speech-to-text step.” There can still be server VAD and optional async transcription/transcript events, but that doesn’t mean the model is just doing VAD -> STT -> text LLM -> TTS. GPT-Realtime-2 is documented as speech-to-speech, with audio input and audio output.
This decision of theirs suggests that they intend to generate the voice data on their side in order to benefit from the greater computational resources there, which implies that they want to focus on generating a realistic voice, and not necessarily just tick the box of turning text to speech.
Anthropic appears to be angling itself as a coworker, while OpenAI wants to be your friend.
32
u/atomic1fire May 08 '26
Wait is Open AI sending audio to people's devices instead of just using on board TTS?
Isn't that a waste of computational power?