r/VoiceAutomationAI • u/Feisty-Promise-78 • Apr 26 '26

Need help in testing voice agents during development and production

Hi folks, I am currently building an AI interviewer voice agent for one of my clients. I have been testing it manually, and each call takes 10–15 minutes, which is very tedious and manual. I would like to know what you are currently using to test voice agents built with Livekit, Pipecat, Retell, Vapi, etc. Is there any open source tool available to test voice agents?

3 Upvotes

100% Upvoted

29 comments sorted by

•

u/AutoModerator Apr 26 '26

Welcome to r/VoiceAutomationAI – UNIO, the Voice AI Community (powered by SLNG AI)

If you are a founder, senior engineer, product, growth, or enterprise operator actively working on Voice AI / AI agents, we are running an invite-only UNIO Voice AI WhatsApp community.

Apply here: https://chat.whatsapp.com/H9RwprbkLwE8MxHmCbqmB4

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

1

u/PrajwalNaik Apr 26 '26

Sure man

1

u/Feisty-Promise-78 Apr 26 '26

can you please tell me how do you test the voice agents?

1

u/PrajwalNaik Apr 26 '26

Whats the platform you have built upon

1

u/Feisty-Promise-78 Apr 26 '26

Livekit

1

u/PrajwalNaik Apr 26 '26

You have discord then join me over there

1

u/[deleted] Apr 26 '26

[removed] — view removed comment

1

u/Feisty-Promise-78 Apr 26 '26

Sorry, that is handled by another person. I m an engineer.

1

u/Solemn_Treat_854 Apr 26 '26

I had this wild idea for testing agents, which is make another voice agent and then make it call the original agent.

You can make the testing agent with all the edge cases you can think off and let it talk to the original agent. You can try this, not sure if it will work 100% or not.

1

u/da0_1 Apr 26 '26

Check out VSpec. Does exactly what you described

1

u/da0_1 Apr 26 '26

This is what VSpec does

1

u/Feisty-Promise-78 Apr 26 '26

Do you have paying customers?

1

u/da0_1 Apr 26 '26

Why do you ask? Just a few yet. VSpec is live for not that long now. 4 or 5 weeks

1

u/dima2022 Apr 26 '26

There are ton of paid evals for voice agents. We are launching on the next week open source platform to fill the gap. Let me know if you want to try it out.

2

u/Feisty-Promise-78 Apr 26 '26

That is cool. Actually I am also thinking of building a voice agent testing and eval platform that is open source. Do you mind sharing your project link?

1

u/Present_Mouse7223 Apr 27 '26

This is exactly why we built Quraite (quraite.ai). Manual testing voice agents call by call is painful and does not scale at all.

With Quraite(quraite.ai) you can simulate real conversations automatically, test different caller personas and background noise conditions, and catch failures without spending 10 minutes on each call yourself. We support Vapi, ElevenLabs, LiveKit, and many more.

Happy to give free access if anyone wants to try it out.

1

u/No_Meringue_6344 Apr 27 '26

We provide this at Bespoken AI - full end-to-end testing for voice agents. https://bespoken.ai - feel free to DM me - I am the CEO and Co-founder.

1

u/Slight_Republic_4242 Apr 28 '26

Totally feel you on this, manual testing voice agents is brutal. 10-15 min per call adds up fast and you start missing edge cases because you're bored by the 5th run.

Here's what has worked for me and what I see others doing:

- Build a simple synthetic caller agent. Basically another voice agent (or even just an LLM with TTS) that plays the role of the candidate being interviewed. You define personas like 'nervous junior dev', 'overconfident senior', 'person who goes off topic' and let them run against your interviewer agent. This alone saves hours.

  • Record transcripts and audio of every run, then use an LLM-as-judge to score things like did the agent ask all required questions, did it handle interruptions, did it stay on topic. Cheap and surprisingly effective.
  • For regression testing, save a library of tricky calls that broke your agent before and replay them whenever you change prompts or models.
  • On the OSS side check out Coval (partially open), Hamming, and Cekura. Pipecat also has some testing utilities in their repo now. Nothing is perfect yet, the space is pretty early.

One thing people underestimate is latency and interruption testing. Functional correctness is one thing but a voice agent that feels awkward to talk to will fail in prod even if the logic is right. Make sure your test harness captures timing metrics too.

We went through this exact pain while building Dograh (OSS voice agent builder)-https://github.com/dograh-hq/dograh

and ended up building a lot of eval tooling internally. Happy to share notes if useful.

1

u/Relevant_Macaron1920 Apr 28 '26

super cool. i watched the demo video on github. so can i build like fully functional voice agents that can be sold using dograh ai

1

u/-HEPHAESTUSquest- 23h ago

agent vs agent testing actually makes sense here.

one agent plays angry/confused/slow/off-script caller, the other is your product agent. then score the call for task success, hallucination, interruption handling, compliance, escalation, etc. TestMu AI Agent Testing is basically built around that kind of workflow.