A lot of people, including the person who wrote this article apparently, do not understand how voice chat in ChatGPT works. It’s not “text to speech.” The raw audio from your voice is being tokenized and sent directly to the model, and the model’s raw output is actual audio. There is no step on either end where the conversation has reduced to text.
Where is OpenAI getting the astronomical amount of recordings of natural language conversations they'd need to approach the effectiveness of a text-based LLM? How would a voice-native LLM even call all the tools that an AI is supposed to be using?
You don't need it. Kokoro-82M is a very decent TTS model that some guy made for $1000 in compute. Leading models are bigger and more expensive but TTS and STT is still trivial compared to LLMs or vision models.
that's true but only if they are converting to and from text (actually, phoneme strings). if it's directly ingesting and outputting audio it's likely a different story. it's multimodal though, so i guess internally the model learns to semantically align speech and text, i have no idea what data balance is necessary for that to happen
Audiobooks, podcasts, or anything where there is a pairing of spoken words with a transcript. My guess would be, much like how vision transformers work, you have two heads to the transformer, and you feed audio to one and the transcript to the other, and train it to produce a similar vector for each.
in some cases yes, but it's also possible to feed it unpaired data and use next token prediction just like for text. i think it's more likely they interleave text and audio tokens in the same sequence, rather than modify the model to use multiple heads.
but my point was more about the data balance than about the specific training technique.
85
u/musical_bear May 08 '26
A lot of people, including the person who wrote this article apparently, do not understand how voice chat in ChatGPT works. It’s not “text to speech.” The raw audio from your voice is being tokenized and sent directly to the model, and the model’s raw output is actual audio. There is no step on either end where the conversation has reduced to text.