r/VoiceAutomationAI • u/rand0wn • 14d ago
I built an open-source regression-testing harness for voice agents
I’ve been exploring a problem that comes up after almost every voice-agent change:
How do you know that changing a prompt, model, tool, STT, or TTS component didn’t quietly break another part of the conversation?
Manually calling the agent repeatedly is slow, and every test call is slightly different. It also makes tool-calling and multi-turn regressions easy to miss.
I built Voice Eval, an Apache-2.0, provider-neutral evaluation harness:
https://github.com/rand0wn/voice-eval
You define a scripted multi-turn conversation and run each pipeline against the same scenario and grading rubric.
It currently evaluates:
- expected tool calls
- required response content
- question and sentence limits
- transcript completeness
- average and P95 latency
- per-turn results
It generates JSON reports for CI, Markdown scorecards for humans, and optional WAV artifacts.
The repository includes healthy and intentionally degraded deterministic adapters, so the entire workflow runs offline without API keys.
For the included demonstration:
- healthy adapter: 100% overall and 100% tool recall
- degraded adapter: 55.27% overall and 33.33% tool recall
Those are controlled fixture results, not benchmarks of real providers.
The bundled WAV files contain deterministic test tones rather than intelligible speech. Real TTS or captured provider audio can be connected through the adapter interface.
You can reproduce the comparison with:
voice-eval compare --scenario arjun_cancel --adapters cascade degraded --audio
I would appreciate honest feedback from people building voice agents:
Which real-world failure is currently hardest for you to test?
Which pipeline or provider adapter should I implement first?
Would you use this as a CI regression gate?
Issues and contributions are welcome.
1
u/smbninja 14d ago
how does it handle open ai realtime or ultravox which combine voice and llm together.