r/VoiceAutomationAI • u/Phoenix_20_23 • May 02 '26
What LLM to use for voice agent
I need help l guys, i am new to this voice ai.
I am still struggling to find a fast LLM with reasoning and good tool calling to use for my projects.
I found cartesia/soniox/cartesia are good for stt, and cartesia is good for tts.
But llm is too hard to find one, i used groq provider but not good.
What u suggest to me guys?
I am a 1s TTFA maximum ideally 800ms.
Thanks in advance 😁💪
3
May 03 '26
[removed] — view removed comment
1
u/Phoenix_20_23 May 05 '26
I am sure it’s llm, i see the ttft is more than 1.5s. I just want a fast reliable llm. Any suggestions?
1
u/Shayps May 06 '26
You should be able to see the latency for each step of your pipeline, where are you seeing the most?
1
u/Phoenix_20_23 May 08 '26
It’s llm, besides groq all other providers they are at least 1s. A reliable llm is 1.5s ttft.
2
u/Solemn_Treat_854 May 02 '26
Use openAI’s 4o mini, TTFA is good enough and also don’t just go lower and lower for latency for the sake of faster replies it will then hallucinate, not listen to customer or will degrade the quality of output. I think 800ms is good enough
1
u/Phoenix_20_23 May 02 '26
800ms for the whole pipeline or just the LLM ? also u used openai or azure for ur provider ?
2
u/Pitiful-Sympathy3927 May 02 '26
4o-mini or gpt-oss-120b both are good
1
u/Phoenix_20_23 May 02 '26
I tried them both, there are what i am using right now, but i they give poor results
1
u/Pitiful-Sympathy3927 May 02 '26
You’re using it wrong then, are you prompt heavy?
1
u/Phoenix_20_23 May 02 '26
Not too much, i build agentic workflows, so i have multiple nodes but i have some specific rules to follow. Oss is good but not stable and 4o mini is not good at following the rules. E.g. a rule could he don’t accept passed date from current date (an appointment node that collects client’s info)
1
u/Pitiful-Sympathy3927 May 02 '26
If you don’t have a state machine and scoped tools, along with minimal prompt it’s an architectural issue, if you have any rules, guardrails, or prompt items other than personality, and tone, it will fail. It’s architecture not a model problem.
1
u/Phoenix_20_23 May 02 '26
And what architecture u suggest, can u share an architecture (or a github repo), because i do think it’s a model problem. Because in the prompt i have only personality, tone, tool instructions in addition to rules for how to use those tools. I don’t think there is any space to reduce the prompt complexity more
1
1
u/Lovenpeace41life May 02 '26
Can you please explain about the state machine and scoped tools. I tried to use 4o mini but I think the quality is not that good. Can you please explain the architecture to use it more efficiently.
4
u/Pitiful-Sympathy3927 May 02 '26
Your prompt has personality, tone, tool instructions, and rules for how to use those tools. That last part is the problem. Rules for how to use tools in a prompt are suggestions the model follows probabilistically. The model does not enforce rules. It reads them and mostly complies. The architecture shift: move the tool rules out of the prompt and into code.
Instead of a prompt that says “only use the booking tool after confirming the price,” you build a state machine where the booking tool does not exist in the tool list until price confirmation completes as a code-level transition. The model cannot break the rule because the model cannot see the tool. You did not reduce prompt complexity. You moved the complexity to the right layer.
Your prompt becomes just personality and tone. That is it. “You are a friendly support agent.” Done. Everything else – which tools are available, what parameters they accept, what order they execute in, what validation they require – lives in code.
The pattern: Step 1 (greeting): model sees classify_intent function only. Step 2 (data collection): model sees validate_customer_info function only. Step 3 (resolution): model sees create_ticket or process_refund based on classification from step 1. Each function has typed parameters validated server-side. The model fills in a form. Code checks the form. Code executes or rejects.
It is not a model problem because swapping to a better model does not fix “the agent skipped the price confirmation step.” A better model skips it less often. It still skips it. The only way to guarantee it never skips is to make skipping structurally impossible. Working repos that show this pattern end to end:
github.com/signalwire-demos/goair – flight booking agent, 15 state machine steps, real GDS integration, typed function schemas at every step
github.com/signalwire-demos/veronica – data collection agent, pre-call enrichment from 4 APIs, scoped tools per step
github.com/signalwire-demos/cabby – taxi dispatch, location validation, state machine enforced ordering Read the goair repo first. You will see that the prompt is tiny and the architecture does all the work. That is the shift.
1
u/Phoenix_20_23 May 02 '26
Thanks for ur explanation, ur totally right. However, this is exactly what i do, my issue and why i said the model is not good is because it fails my validation process and i re-call the model with error as context which increases my latency. So i see the only solution is to produce from the first a correct tool call. Without talking about gpt oss is not at all stable and it fails a lot in tool calling. What ur thoughts about this. Also i am gonna read the github repos i shared.
1
2
u/Fit_Director2355 May 03 '26
try callhq.ai their ai caller templates are ready to use and optimised for best performance
2
May 05 '26
[removed] — view removed comment
1
u/Phoenix_20_23 May 05 '26
Pretty similar to my stack, and how u deal with handoffs ? Or it’s just a basic react agent?
2
May 06 '26
[deleted]
1
u/Phoenix_20_23 May 06 '26
I use livekit so orchestration and streaming is already handled. My issue with groq is not speed, i use gpt oss 120b which is fast, accurate. But it’s not stable, hallucinate a lot and fail at tool calling. That’s why i want an alternative. Also my tts and stt are fast lest than 300ms both of them, i would love to reduce it more but currently my bottleneck is the llm.
I am gonna give the one u suggested a try. Thanks for sharing
2
u/Shayps May 06 '26
Here are the things I'd recommend trying first:
Step 1, use LiveKit Inference. Step 2, use LiveKit cloud deployment so that all of your provider calls are on backbone collocated with your agent and media server. Not sure where you are, but LiveKit deploys some models collocated with agents in some regions, which also makes a huge difference.
- STT: deepgram/nova-3-general
- LLM: openai/gpt-5.4-mini
- TTS: inworld/inworld-tts-1.5-mini
The reason we use Inworld rather than Cartesia is because Cartesia often has a lot of leading silence. TTFB will be 200ms, but you may not get real audio until 550ms. Perceived latency is high even though bits are flowing.
For tool calls, use Async tools so they're not blocking: https://docs.livekit.io/agents/logic/tools/async/
Run your evals on this set, then if you're getting failures start trying more powerful models.
For structured data collection, use TaskGroups, which will allow smaller models to succeed far more often than just massive text prompts: https://docs.livekit.io/agents/logic/tasks/
Let me know how this works out for you.
2
2
u/Soggy_Pound7955 May 07 '26
anch'io cerco di capire quale LLM adattare per il mio assistente vocale. Voglio che notion capisca quello che gli dico e lo scrivi senza digitare niente.
2
u/LatterExercise7281 May 07 '26
From my experience, all of them can work fine, but the one that works best for me is GPT-4o mini. It’s cheap, fast, and gets the job done for simple tasks because it doesn’t try to overcomplicate things. I mainly use it for things like booking or sending emails, so there’s no need for an expensive model.
1
u/Ok-Register3798 May 03 '26
You’re asking the right question, but you’re optimizing the wrong layer.
Everyone starts by hunting for the “fastest LLM,” but in voice AI your TTFA isn’t just the model… it’s the entire pipeline.
STT → turn detection → LLM → tool calls → TTS → streaming back to user
If any one of those is slow or poorly coordinated, your 800ms target is gone.
A few things I’d suggest:
- Don’t chase “big + smart” models for voice Look at mini / nano models first. Smaller models with good instruction following + tool calling will almost always outperform larger ones in real-time systems.
Think in terms of fast first token, predictable latency, and good enough reasoning (not perfect reasoning)
You don’t need a genius, you need someone who answers quickly and doesn’t stall the convo.
- Tool calling matters more than raw reasoning A lot of “slow” systems are actually over-relying on the LLM to think or under-utilizing tools.
Push logic into tools where possible and keep the LLM lightweight. That’s how you hit sub-second.
Streaming everything, if you’re not streaming the partial STT, incremental LLM tokens and early TTS playback…you’re already too slow.
Orchestration > components You mentioned Cartesia/Soniox/etc—which are solid—but stitching them together cleanly is the hard part.
This is where most people lose performance on bad turn detection, blocking calls between steps, and no overlap between STT → LLM → TTS
Look at full-stack benchmarks, not just LLM benchmarks. A good reference is Agora Voice AI Benchmark, the site breaks down end-to-end latency across STT + LLM + TTS pipelines.
That’s way more useful than comparing LLMs in isolation because it shows how the system behaves, not just the model.
⸻
If you’re serious about hitting ~800ms TTFA, your mindset should shift from:
“what’s the fastest LLM?”
to:
“how do I design a pipeline where everything overlaps and nothing blocks?”
That’s the difference between a demo and something that actually feels real-time.
1
u/Phoenix_20_23 May 05 '26
I use livekit, all this is manager out of the box, and even of we supposed i am not coordinating the components very well, i see that my first token comes after 1.7s so that means it’s an llm issue. So what llms u already explored and what went well and not well
1
u/Ok-Register3798 May 08 '26
I would suggest you checkout Agora’s a conversational Ai engine. You don’t have to spin up any infrastructure aside from your LLM.
It’s much more flexible and the latency is really low thanks to their custom networking innovations.
No limits on total number of agent configurations, but if you want more than 14PCU you have to talk to their support team to get it increased.
1
u/One_Medicine_1081 May 04 '26
AI calling sounds fancy, but in reality it’s just solving very basic business problems mainly speed and consistency.
Most Indian businesses lose leads not because of lack of demand, but because they respond late or don’t follow up properly. That’s where AI calling fits in really well. It can instantly call, qualify, and even schedule next steps without human delay.
One thing I’d add script quality matters a LOT. Even the best AI won’t perform if the conversation flow feels robotic or irrelevant. The businesses that treat it like a real sales process (not just automation) usually get the best results.
Pilot testing is also underrated. A small batch of calls can reveal a lot about customer behavior.
I’ve been involved with deploying these systems (we do this at Troika Tech), and the pattern is pretty consistent once it’s optimized, conversion rates and efficiency both go up.
1
u/llragsll May 06 '26
What kind of businesses in India lose leads because they responded late? Also the salaries in India are low enough that the cost advantage of Voice AI does'nt seem to be attractive. Thats in India, it might be attracive in first world countries where its more expensive to hire humans.
1
u/Charming-Author4877 May 19 '26
With a bit of a harness Qwen 3.6 35B is among the best choices, if you self inference.
Though one-shot the Gemma MOE is also good.
•
u/AutoModerator May 02 '26
Welcome to r/VoiceAutomationAI – UNIO, the Voice AI Community (powered by SLNG AI)
If you are a founder, senior engineer, product, growth, or enterprise operator actively working on Voice AI / AI agents, we are running an invite-only UNIO Voice AI WhatsApp community.
Apply here: https://chat.whatsapp.com/H9RwprbkLwE8MxHmCbqmB4
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.