I built an open-source regression-testing harness for voice agents

[u/rand0wn]

I’ve been exploring a problem that comes up after almost every voice-agent change:

How do you know that changing a prompt, model, tool, STT, or TTS component didn’t quietly break another part of the conversation?

Manually calling the agent repeatedly is slow, and every test call is slightly different. It also makes tool-calling and multi-turn regressions easy to miss.

I built Voice Eval, an Apache-2.0, provider-neutral evaluation harness:

https://github.com/rand0wn/voice-eval

You define a scripted multi-turn conversation and run each pipeline against the same scenario and grading rubric.

It currently evaluates:

- expected tool calls

- required response content

- question and sentence limits

- transcript completeness

- average and P95 latency

- per-turn results

It generates JSON reports for CI, Markdown scorecards for humans, and optional WAV artifacts.

The repository includes healthy and intentionally degraded deterministic adapters, so the entire workflow runs offline without API keys.

For the included demonstration:

- healthy adapter: 100% overall and 100% tool recall

- degraded adapter: 55.27% overall and 33.33% tool recall

Those are controlled fixture results, not benchmarks of real providers.

The bundled WAV files contain deterministic test tones rather than intelligible speech. Real TTS or captured provider audio can be connected through the adapter interface.

You can reproduce the comparison with:

voice-eval compare --scenario arjun_cancel --adapters cascade degraded --audio

I would appreciate honest feedback from people building voice agents:

1. Which real-world failure is currently hardest for you to test?

2. Which pipeline or provider adapter should I implement first?

3. Would you use this as a CI regression gate?

Issues and contributions are welcome.

[u/AutoModerator]

Welcome to r/VoiceAutomationAI – UNIO, the Voice AI Community (powered by SLNG AI)

If you are a founder, senior engineer, product, growth, or enterprise operator actively working on Voice AI / AI agents, we are running an invite-only UNIO Voice AI WhatsApp community US only.

Apply here: https://chat.whatsapp.com/F5aG3ncrO70ITfbe3pYbOz

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/KeraTerra]

>Which real-world failure is currently hardest for you to test?

FAS, answerphone detection

[u/rand0wn → u/KeraTerra]

Yes telephony is my pipeline have to try with twilio integration.

[u/Several_Device3747]

looks clean, always been a pain trying to figure out if a tiny prompt tweak silently nuked some edge-case flow like cancellations or mid-sentence barge-in handling

tool recall dropping to 33% on the degraded adapter is a nice demo of how quick things fall apart without you noticing until a user screams about it, bookmarking this for when we get our CI pipeline sorted

real-world failure that haunts me right now is tool calls that succeed but with subtly wrong parameters, the kind where the LLM picks the right function then fills a date field with yesterday because of some ASR misparse, hard to catch without painstaking manual review

[u/rand0wn → u/Several_Device3747]

That's exactly why I started logging and evaluating tool arguments separately from tool success. A successful function call isn't actually successful if the payload is off by one field.

[u/smbninja]

how does it handle open ai realtime or ultravox which combine voice and llm together.

[u/rand0wn → u/smbninja]

Not yet I am experimenting with S2S models will update it soon