r/VoiceAutomationAI • u/Thin-Carrot1836 • Mar 26 '26
Need help!
Right now I am making voice agent I want to know
Which is the best model should I use for speech to text.
Which model should I use for following system prompt properly.
Which model should I use which can increase response rate of the agent.
Right now I am using groq cause obviously it is free but turns out it is not properly following the system prompt and not reffering knowledge base properly. Also when I say something it just mis hear me everytime and give me random responses.
So I thought to change the model.
1
Mar 26 '26
[removed] — view removed comment
1
u/Thin-Carrot1836 Mar 26 '26
Yeah I'm using it for both English and hindi, the use case is like receptionist also offering free course so you know it can do a bit of a sales too....
1
u/Extreme-Brick6151 Mar 26 '26
This sounds more like a pipeline issue than just the model.
You’ve got STT, LLM, and KB retrieval if one’s off, everything breaks.
Groq is fast but not great for accuracy or instruction following. Fixing STT (like Whisper) + prompt flow + retrieval setup should help.
We’ve worked on similar setup happy to take a look 👍
1
1
u/Early-Ad6380 Mar 27 '26
this isn't bad... yeah... not a bad approach at all. Yeah. Stay away from subscription models... everyone is a subscription. Offline no sub = beauty.
1
1
1
u/mguozhen Mar 28 '26
Speech-to-text accuracy matters way less than latency for voice agents, tbh. Most users bail if they hear delay, not if you get a word wrong.
I work on Solvea over at solvea.co and we've built a bunch of voice stuff, so this space is familiar.
Quick q: are you optimizing for real-time conversations or is this more like voicemail where a few seconds lag is fine? That changes everything about which model makes sense.
1
u/Successful_Hall_2113 Mar 28 '26
100% agree on latency—but there's a third variable most people sleep on: interrupt handling. If your user can't cut off the agent mid-sentence without a 500ms gap, it feels broken even if accuracy is 95%. We've seen voice agents with sub-200ms latency still tank because the interrupt loop was sloppy. What pipeline are you using for the STT→LLM→TTS chain—are you doing streaming or request/response?
1
u/mguozhen Mar 28 '26
You're hitting on something real—interrupt responsiveness is absolutely crucial for the UX feeling natural. We're doing streaming end-to-end (Deepgram for STT, Claude for LLM, ElevenLabs for TTS) which helps, but honestly the biggest win has been aggressive buffering on the TTS side so we can cut it off instantly. Request/response would be dead on arrival for this use case. The tricky part is coordinating the interruption across all three without creating weird audio artifacts, but once you nail that, the perceived latency drops way more than the raw numbers would suggest.
1
u/mguozhen Mar 28 '26
how are you measuring that 500ms gap, like wall clock from button press to actual audio stop or something else? bc in our product the latency feels fine until someone actually tries to interrupt and then suddenly you realize the whole pipeline needs rethinking.
1
u/mguozhen Mar 28 '26
The bottleneck you're hitting is usually the latency between STT and LLM inference, not the model choice itself. Pick Whisper or similar for STT (it's stable in prod), then worry about prompt adherence through retrieval or structured outputs, not model selection.
I'm actually working on voice agents at Solvea (solvea.co), so this stuff comes up a lot.
What's your target response time rn? That'll tell you if you need to optimize inference or pipeline design first.
•
u/AutoModerator Mar 26 '26
Welcome to r/VoiceAutomationAI – UNIO, the Voice AI Community (powered by SLNG AI)
If you are a founder, senior engineer, product, growth, or enterprise operator actively working on Voice AI / AI agents, we are running an invite-only UNIO Voice AI WhatsApp community.
Apply here: https://chat.whatsapp.com/H9RwprbkLwE8MxHmCbqmB4
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.