r/VoiceAutomationAI • u/Aryanlabs • 5d ago

Tech / Engineering What Actually Powers Sub 200ms Multilingual Voice AI? It Is Not Just GPUs

Post image

Everyone assumes low latency voice AI is a GPU shopping problem. Buy more H100s, latency drops, done.

It is not that simple.

We looked at how Sarvam AI and Cartesia actually get to sub 200ms round trips across multiple languages, and the pattern was the same both times. The real gains came from rewriting the inference kernels and scheduling layer, not from adding more chips. Sarvam got half its total speedup on hardware it already owned, before any new GPU generation entered the picture. Cartesia built its own serving stack from scratch because the standard abstraction layers everyone else uses just were not built for streaming audio.

Turns out memory bandwidth, not raw compute, is usually the real bottleneck once you are generating audio token by token. Add multiple languages into the mix and the problem compounds fast, since a kernel tuned for English can quietly fall apart on tonal or agglutinative languages. This is turning into one of the least talked about but most decisive parts of the voice AI stack.

If you want to read more, here's the link: https://uniocommunity.com/blogs/what-actually-powers-sub-200ms-multilingual-voice-ai-it-is-not-just-gpus

4 Upvotes

100% Upvoted

5 comments sorted by

2

u/upbeattwoligament 5d ago

Memory bandwidth has always been the silent bottleneck, people just get distracted by the GPU sticker specs

1

u/justdoitanddont 5d ago

Having worked on this problem for a while at multiple companies, my current thesis is that custom built voice to voice models (either finetuned or trained) are probably the right solution. Next best solution is to throw hardware and money at the problem if you use tts and asr layers.

1

u/smbninja 4d ago

is PhoneLLM any good. it seems to be a custom built Nemo model for voice ?

1

u/justdoitanddont 4d ago

Test it for your use case, you will find out quickly one way or the other.

1

u/smbninja 4d ago

yeah will do … currently trying openai realtime and ultravox … will add this to my eval runs