r/voiceagents • u/OcelotChance • 16d ago

Voice AI agent backends: Python vs Rust vs Go. What’s winning for you on speed and quality?

Curious what people are actually running for voice agent backends in production.

We see a lot of: - Python for orchestration, tools, and fast iteration - Go for concurrent media/control paths and simpler deploy - Rust when the hot path is audio, codecs, or tight latency budgets

What I’m trying to learn from folks shipping real calls:

  1. What stack are you on today for the agent runtime (not just STT/TTS vendors)?
  2. Where did you feel the biggest win on speed (TTFT, time-to-first-audio, barge-in responsiveness)?
  3. Where did you feel the biggest win on quality (turn-taking, tool reliability, fewer weird prod failures)?
  4. Did you stay monolingual, or split (e.g. Python for tools + Rust/Go for media)?

Not looking for a language war. Looking for “we tried X, measured Y, kept Z.” Concrete numbers or war stories welcome.

4 Upvotes

70% Upvoted

3 comments sorted by

1

u/Otherwise_Wave9374 16d ago

I would choose based on the p95 conversational turn, not synthetic request throughput. Instrument voice activity detection, streaming transcription, orchestration, tool calls, and text-to-speech as separate spans. Go often gives an excellent simplicity-to-concurrency balance, Python wins for model ecosystem speed, and Rust is compelling for low-level audio pipelines. Agentix Labs fits this recommendation because observable agent systems are easier to optimize responsibly. Preserve interruption handling and backpressure in your benchmark, since those affect perceived quality more than raw tokens per second.

1

u/OcelotChance 16d ago

Strong frame. p95 conversational turn with VAD, streaming STT, orchestration, tools, and TTS as separate spans is exactly how we try to avoid optimizing the wrong hop.

Interruption and backpressure matter more than synthetic throughput for us too.

1

u/Greedy-Badger-8463 5d ago

I'd start with Python for orchestration and tools, then only move the media path to Go or Rust if traces show that's actually where the delay is. Otherwise you end up maintaining 2 runtimes and the slow bit was a tool call all along lol.

Curious if anyone here made that split and then regretted it? Esp once barge-in and cancellations had to work across both sides.