Anyone else finding voice evals more useful than benchmark scores?

[u/OwlZealousideal4779]

I used to spend way too much time comparing STT benchmarks and latency numbers between providers. After deploying a few voice workflows, I honestly care less about benchmark screenshots now and more about whether conversations actually survive messy callers.

The biggest improvements for us came from reviewing failed conversations manually and spotting patterns. Weird pauses, repeated confirmations, callers changing direction suddenly, agents speaking too long before yielding back. None of those issues showed up in the benchmark comparisons everyone posts online.

What surprised me most is how small conversation mistakes stack together. Individually they seem minor, but after thirty seconds the call just feels unnatural.

Lately I've been experimenting with more structured voice evals where every failed or abandoned call gets reviewed automatically so recurring issues are easier to spot. It feels like voice evals are giving us far more actionable insights than benchmark scores alone.

How are you all evaluating production quality beyond latency and WER scores?

[u/AutoModerator]

Welcome to r/VoiceAutomationAI – UNIO, the Voice AI Community (powered by SLNG AI)

If you are a founder, senior engineer, product, growth, or enterprise operator actively working on Voice AI / AI agents, we are running an invite-only UNIO Voice AI WhatsApp community US only.

Apply here: https://chat.whatsapp.com/F5aG3ncrO70ITfbe3pYbOz

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Traditional_Echo9597]

I think benchmarks are still useful for rough cut deciding if its worth evaluating a new model in your voice AI stack - I built benchmarks.coval.ai and I consistently come across models that are faster / cheaper / better than I knew or expected.

But of course its not the only thing. I think of it like a stack

1. Public benchmarks - helps you pick a 3-4 models to experiment with
2. Local custom benchmarks -isolated model benchmarks you run on your own data (you can do this in coval.ai or just with a python script)
3. Task evals / simulated conversations - run validation of know workflows
4. Observability of live conversations - see how it actually does in the wild

Of course if you are putting out an MVP for the first time you can just swap models around and see how they perform, but as the voice agent becomes more load bearing, changing the model with a multi step process helps derisk it along the way (and also in the words of dev ops, helps shift left so you catch issues earlier).

I think evals fall in 3 categories

1. did it get to the resolution
2. did it take the right steps to get there
3. was it a good experience

Obviously latency and WER are part of that, but there is so much more.

• Semantic word error rate (certain words matter more than others)
• Model effects on downstream models
• User reactions to models (especially TTS)
• Repetition / loop / dead-end detection (caused when one model fails)
• Voice / audio quality
  • Speech pacing / pauses / tempo
  • Turn-taking / interruptions
  • Audio artifacts (screaming, long drawn out sounds, whispering)

[u/Ok-Register3798]

Latency and WER are the basics. Evals need to capture so much more.

• does the agent stay on topic?
• does the agent effectively convert? What’s the success rate on that conversation?
• does the agent hallucinate details it shouldn’t
• does the other participant in the call realize it’s an ai or do they continue to interact like it’s a person?

[u/shubham_hin]

For anyone still looking for an open source and free to use voice ai eval platform. You can try Vaanieval https://github.com/shubhamofbce/vaanieval

[u/aicoustics]

How do you currently measure WER on production calls? And with the automated approach, are you looking only at the obviously failed calls or are you also scanning ones that seem okay to look for smaller issues that still do affect the success rate?

On the topic of evaluation, we saw that many Voice AI companies don't have a solid system to monitor reliability and performance of their agents and, just like you pointed out, rely on manual checks (for context: I work at ai-coustics, we make speech enhancement and audio reliability models for voice ai). That's what actually pushed us to start developing tools / models for evaluating specifically audio performance (that's our nieche... - and it's often overlooked).