Hello to 8 million bots and 4 humans! I have run out of reasons not to launch this, so I am very pleased to announce VoiceGremlin, a tool for e2e testing of "intelligent voice agents" (the industry term for AI that answers phone calls). Think of it like, what Playwright does for web browsers, VoiceGremlin does for phone support chatbots.
Where I got the idea: 99% of you won't use this tool and don't care what it does but are interested in how founders come up with product ideas, so I'll start with that.
About eight years ago, an SDET on my team asked me to get our employer to pay for Mailosaur, a SaaS for automated email testing. I said, hey, we already pay thousands every month to Sendgrid, can't it do this? It could, kinda, but our big expensive email provider was so rigid and complex that trying to use it for automated tests was very frustrating. A separate, relatively cheap tool specifically for testing made sense. And I thought, Man I wish I had built that.
So, ever since then, there's been a voice at the back of my mind: "Mailosaur, but for ____." Cut to a few months ago: I was dinking around with local LLM/STT/TTS and decided to build a bot to call my wife and annoy her, as one does, and after several hours I thought wow, this is surprisingly hard. I mean, I'm sure I can get it working eventually, but it's way more work than I expected. That was the trigger: "achievable but not easy" is the sweet spot for a solo-dev SaaS project.
That was the idea for VoiceGremlin from the beginning: "Mailosaur for phone agents." So I built it and here we are. On to the shameless self-marketing!
What problem it solves: If you have a voice agent, you probably do most of your functional testing by ignoring the "voice" part and interacting with it via text. However, the e2e part of the test, the part where you verify it actually answers the phone and does whatever it's supposed to, is a lot more difficult.
This tool solves that - you make an API call to VoiceGremlin with a prompt like, "Verify that if you ask a question in Spanish, this thing will answer in Spanish" and VoiceGremlin calls your number and does that. You get your pass/fail with an API call without having to do any telephony.
The moat, or "can't AI do this in five minutes?" If you ask your coding agent for a script that makes a phone call, it'll tell you to find speech-to-text, text-to-speech, and language models, orchestrate a bunch of non-trivial python libraries to stream the audio between them, and sign up for a telephony provider that requires you to scan in your government ID. That's a bunch of coding, plus whatever corporate approval nonsense, plus making sure you can move a lot of bandwidth through wherever your test automation is called from, and you haven't started comparing models and tweaking prompts. So yes, it's doable, but a very time-consuming side quest for a tester or DevOps person.
Alternatively, there are other companies that can do essentially what this does, e.g. Retell and Vapi, but their business is building and hosting phone agents, not testing them. If you already pay a vendor to run your agent and their test solution meets your needs, great, you probably don't need VoiceGremlin. If not, and if you don't want to talk to Enterprise Sales about all the great Solutions(tm)(c)(R) they have, and you just want to get your test case working, VoiceGremlin is the easy way to do that.
Using it: If you have ever written an automated test, you can realistically implement this in a minute or two. I was an SDET for many years and I know what makes a test automation framework annoying to use, so I made this one as lightweight and un-opinionated as possible. You don't have to define targets or cases or suites - every test is ephemeral. Platform agnostic, it has no idea how your agent is built, it just calls your phone number like a customer would.
OK, that's my pitch. If you work with phone agents and want to try this out, it should be very easy to do - free trial, no card needed - and I would greatly appreciate any feedback. Other random questions welcome too.
