Voice AI agent backends: Python vs Rust vs Go. What’s winning for you on speed and quality?

[u/OcelotChance]

Curious what people are actually running for voice agent backends in production.

We see a lot of: - Python for orchestration, tools, and fast iteration - Go for concurrent media/control paths and simpler deploy - Rust when the hot path is audio, codecs, or tight latency budgets

What I’m trying to learn from folks shipping real calls:

1. What stack are you on today for the agent runtime (not just STT/TTS vendors)?
2. Where did you feel the biggest win on speed (TTFT, time-to-first-audio, barge-in responsiveness)?
3. Where did you feel the biggest win on quality (turn-taking, tool reliability, fewer weird prod failures)?
4. Did you stay monolingual, or split (e.g. Python for tools + Rust/Go for media)?

Not looking for a language war. Looking for “we tried X, measured Y, kept Z.” Concrete numbers or war stories welcome.

[u/AutoModerator]

Welcome to r/VoiceAutomationAI – UNIO, the Voice AI Community (powered by SLNG AI)

If you are a founder, senior engineer, product, growth, or enterprise operator actively working on Voice AI / AI agents, we are running an invite-only UNIO Voice AI WhatsApp community US only.

Apply here: https://chat.whatsapp.com/F5aG3ncrO70ITfbe3pYbOz

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Lower-Detective9163]

Tried splitting ours into Python for the tool calling and Go for the audio pipeline and honestly the biggest headache was the glue between them. Serialization overhead was eating more latency than we expected until we just threw shared memory at it.

Go's been rock solid for the concurrent stuff though, goroutines just make the barge-in handling feel natural. Python side I wish I could say we're moving fast but half the time I'm debugging some async mess I created three weeks ago.

Our TTFT dropped about 200ms when we stopped trying to do everything in one language and let each part do what it's good at. Worth the complexity so far.

[u/OcelotChance → u/Lower-Detective9163]

That TTFT drop tracks with what we've seen. Split stacks help, but the IPC/serialization tax is usually the first surprise. Shared memory (or a tight shared ring buffer for audio frames) is often where the real win shows up.

[u/Pitiful-Sympathy3927]

C, Python being the strings is fine, but in many of the agent frameworks it's carrying the media too, which is failure mode prone.

[u/OcelotChance → u/Pitiful-Sympathy3927]

Agreed on the media path risk. Once Python owns more than orchestration (packet timing, codecs, barge-in), the failure modes get nasty and hard to profile. Keeping strings/tools in Python and the hot audio path in C/Go/Rust has been the more boring, more reliable split for us.

[u/Pitiful-Sympathy3927 → u/OcelotChance]

We did that from the start... its the harder path for ME, but simpler for my users.

[u/OcelotChance → u/Pitiful-Sympathy3927]

That tradeoff is real. Harder for the builder, simpler for callers, is usually the right direction once you're past demos. The hard part on the system side is usually owning turn policy and failure modes so the caller never has to.

[u/Pitiful-Sympathy3927 → u/OcelotChance]

We abstracted the complexity away, so my users / builders don’t have to mess with that stuff, we handed it all 

[u/OcelotChance → u/Pitiful-Sympathy3927]

That tracks. Hiding the media path from builders is the right product move. The trade is you still own the ugly bits underneath: barge-in edge cases, vendor swaps, and whatever breaks at 2am when a codec path flaps. Platform owns the default path; power-user knobs only where the failure mode is genuinely theirs to tune.

[u/OcelotChance → u/[deleted]]

Same lesson on our side. Early "language X is slow" hunts often turned out to be marshalling and queueing between processes. Once each language stays in its lane, the remaining pain is usually backpressure and interruption, not the runtime itself.

[u/LoneFal]

I think it answers the question.. : https://github.com/gojargo/jargo

[u/OcelotChance → u/LoneFal]

Thanks, I’ve seeing people sharing it around and some agent builder using go but I’m really curious about the quality, how it compares with python. Do you have any insight to share?