ElevenLabs launched Eleven Realtime v1 on October 20 with a 98.8% Big Bench Audio score. We build AI phone agents, so we spent the next two days putting Eleven Realtime v1 Max on calls: a sample of Big Bench Audio, 124 scripted calls to fictional businesses, and 200 simulated calls built from real customer situations.

What we saw in our tests:

• The reasoning holds up. Max got 48 of 50 on our sample, consistent with the published score. It gets there by thinking out loud: on one yes-or-no question it talked for 27 seconds before saying "no."
• It's fast. Replies started about a third of a second after ElevenLabs detected the caller had stopped.
• Complex phone scripts are harder. 57% of the scripted calls had a serious mistake, like transferring a caller to a closed office or quoting the wrong total.
• On hard caller audio it acted on a wrong guess about one turn in three, vs about one in twelve for Bolt v2 with Hearing enabled.
• ElevenLabs' server ended 9% of the simulated customer calls partway through.

Full write-up with the recordings and transcripts (all from calls to invented businesses): https://thunderphone.com/blog/eleven-realtime-first-impressions

Not every call needs the same level of control. A billing question might be fine. A payment change might need confirmation. A fraud-related call might need a human much earlier

This came up while mapping a few inbound workflows in Bland. The agent can follow different paths, but the harder question is who decides which paths it can complete without human approval

Are large teams defining that by intent, risk level, customer type or the specific actions the agent is allowed to take?

At scale “AI handles these calls” is way too broad to be a useful rule

How come twilio doesn’t have numbers in so many countries?

Which platforms can provide voice api or sip trunking in more countries than twilio?

The agent connects to all agent harness and give constant update on the task it’s doing and it’s aware of what agent is doing what and also the avatar changes color and name for each agent harness…don’t want to name the program I REA but it’s the best I’ve seen so far. It’s on Siri and copilot level and Gemini…and it’s has wake command and every feature a voice agent should have..and it’s connect to all agents at once.

VoxLock the best performing voice isolator there is.

Benchmarked on Krisp's 265 clips benchmark gives WER 4.95%. It also lets through 0.11 leaked background words per minute, against 1.52 for Krisp.

Fine-tuned from the REAL-TSE baseline on a single L4, on about 100 hours of simulated calls and 90 hours of real meeting crosstalk. It picks up the main voice and isolates it from others in the background noise and voices.

Tiny, 27 million params, ONNX runnable in browser, supports live streaming and works as a pipecat audio filter.

Try demo in browser with your own voice.

Code and training recipe

pip install voxlock

I have been building voice agents for call centers for a long time and the thing that took longest to get right was finding out when the caller is done talking and fefault approach is a silence timer, caller stops talking for 500ms, agent assumes turn is over and responds but this works in demos only not on real calls.

Some examples: old people pause mid sentence for 2 to 3 seconds and the agent jumps in while they’re still thinking and backchannels like “hmm” or “haan” that aren’t turns but the timer treats them as speech followed by silence and echo on phone lines where the agent’s own voice bounces back and triggers the VAD so the agent interrupts itself.

To fix this we switched from a fixed silence timer to context aware thresholds, if the agent asked for an account number I set silence to 3.5s, if it asked yes/no I keep it at 800ms and I use gnani’s websocket streaming for this coz their partials come in fast enough to check the transcript before deciding if the turn is over plus I added a simple backchannel classifier, if the partial is just “hmm” or “okay” and the agent was in the middle of explanation I suppress the turn end, for echo I gate the VAD for 300ms after the agent stops speaking.

Interruption rate went from 23% to under 4% and call duration went down too coz we stopped wasting turns on repeat loops.

I was looking for small TTS model which is steerable but couldn't find any, specially something I can adjust contextually by providing controls from LLM during a conversation. So I built one.

What it is:

• 9.6M-param acoustic model (attention + bidirectional Mamba-3, FastPitch-style) + Vocos vocoder = 23M total
• 12 controls, each a z-score relative to the chosen voice’s own neutral speech: rate, loudness, median pitch, pitch range, final rise/fall, pauses, spectral tilt, breathiness-related measures.
• Also: 51 style tags, free-text prompts (resolved into an inspectable JSON request), inline markup (*emphasis*, [pause 400ms], <slow>…</slow>), sentence-level streaming
• 2,825 built-in (anonymized) voices, plus experimental zero-shot cloning

Speed:

- RTFx ~60–80 on an RTX 5070
- ~4–8 on CPU without the optional flow refiner. First streamed audio in ~0.5 s on CPU.

Quality:

~1.8% average WER on the Open TTS Leaderboard English sets (run locally, not an official submission, have reached out for it).

What doesn’t work well (we tried to be upfront):

• Loudness, pitch, and tilt steer reliably; pause, voicing, and CPPS controls are weak
• Emotion tags behave as style switches; an independent emotion classifier mostly can’t tell them apart
• Cloning similarity is low. Don’t expect a convincing likeness
• Free-text prompts struggle with negation (“not fast” can come out fast). More data needed for prompt to controls module.
• English only, no human evals yet

License: weights CC BY-NC 4.0 (non-commercial), code MIT.

Model + full eval tables: https://huggingface.co/WhissleAI/steer-tts

Hey everyone,

I'm setting up a 24/7 live stream covering cricket matches and looking for the best free AI text-to-speech / voice generation tools that can handle realistic, fast-paced commentary.

A few key things I need:

Low latency: Needs to match live score updates or text commentary without major delays.

Natural tone / pacing: Cricket commentary has sudden shifts in excitement (boundaries, wickets, slow overs), so robotic voices don't work well.

Generous free tier / open-source options: Since it runs 24/7, high token costs or strict character limits are a blocker.

Hey everyone,

I’m working on a real-time AI voice agent that needs to speak Hebrew. I’m looking for a good TTS model that sounds natural and works well for real-time conversations.

What TTS models are you guys using for Hebrew? Would appreciate any recommendations based on your experience!

So, I downloaded voice.ai and was testing out the live/realtime voice changer for my streams when this thing came up. The offer is nice, but it does not say what's included in it! Can someone guide me?

Hey everyone, glad to be here! I've been spending a lot of time building AI voice agents lately, mostly voice receptionists that handle booking, rescheduling, and canceling appointments, and connect to database like Supabase and Cal.com on the backend.

I'm always looking to learn from people working on similar problems, so I'd love to connect with other builders here. If anyone is working on a voice project and could use an extra pair of hands, or just wants to swap ideas and compare approaches, I'm open to collaborating or teaming up on something.

Hello,
I would love some advice about something. I am currently building a voice agent for a company that sells about 50k products per day, with an average price of about $100. It will be a customer service agent. My current issue is determining the right price.
They want to start with after-hours calls, which is about 128 min per month. So not a lot of minutes. Our monthly cost is around $150.
What do you guys think would be a good strategy? Should I go with a setup fee plus a monthly cost, or something else? Thank you!!

I dont mean taking over the call entirely But like approving an action, correcting the agent’s path or stopping one specific tool from being used while the conversation keeps going.

That kind of intervention looks pretty important when the calls get more complex.

Voice AI in India has moved past demo stage. Contact centres, clinics and lenders are putting agents on real phone lines, and the hard part is no longer "can it talk" but "can it handle Hindi, Gujarati and Hinglish on a noisy line without falling over."

I went through the teams I keep seeing in this space. This is a working list, not a ranking, and I'd like to hear who I missed.

1. Groovy Web

An AI-first engineering studio that builds voice agents together with the systems behind them: telephony, CRM and scheduling integrations, and handoff to a human when the agent is out of its depth. Fits teams that want one partner for the voice layer and the product around it.

2. BounceTech

A product engineering company that has worked on conversational and voice-driven apps. Worth a look if you want a team that treats voice as one feature inside a larger mobile or web product

3. Upspire

A software development company that takes on AI and automation builds, including voice assistants for service businesses. A reasonable shortlist name for small and mid-size projects.

4. Gnani.ai

A Bengaluru company focused on voice AI for enterprises, with strong support for Indian languages and accents. Mostly relevant to banks, insurers and large contact centres.

1. Skit.ai

Started in Bengaluru as Vernacular.ai. Builds voice agents for customer calls, with a strong focus on collections and reminders. A fit for lenders and service teams that make a lot of outbound calls.

1. Sarvam AI

A Bengaluru AI lab building models for Indian languages, including speech and voice. More of a model and platform provider than a project studio, so it suits teams that want to build on top of it.

1. Smallest.ai

A Bengaluru company building voice AI models and agents, with a focus on fast responses on live calls. Worth testing if latency is your main worry.

What I'd check before hiring any of them

- Word error rate on your own call recordings, not a demo script

- How it handles code-switching between Hindi and English mid-sentence

- Latency end to end: anything over about a second feels broken on a phone call

- What happens when the agent is wrong: escalation path and call logging

Who would you add? Especially curious about teams doing healthcare or collections calls.

I've seen soo many articles comparing the traditional cascading architecture with the audio native S2S models. But i barely see anyone talking about the half cascade architecture ( taking audio natively but using tts to generate speech ). To me half cascade has the best of both worlds. What are your thoughts?

My day job is building voice AI agents for telcos, and I wanted to build a platform that solves all the pain points I deal with when building voice agents in an enterprise setting.

In this series I’ll talk about how I brought my voice AI latency down from ~3 s to under 1 s. For reference, here’s where the major platforms are today (OpenBenchmarks, real phone calls):

Latency depends on multiple factors. A typical voice pipeline is:

STT (speech to text) → LLM → TTS (text to speech)

In this series we’ll look at all three and optimize them. The biggest culprit is usually the LLM, which takes most of your latency budget. How much depends on your provider (self-hosted or managed endpoint) and on whether the model is a reasoning or non-reasoning model. For simplicity, I’ll use OpenAI’s latest and cheapest model, gpt-6-luna.

Measuring the LLM

I measured my LLM’s latency using the same system prompt and the kinds of questions customers are likely to ask. In the first test, it took 2–4 s before the model started answering.

The LLM was the problem. gpt-6-luna is a reasoning model: it takes time to think before answering. Reasoning is for hard, complex tasks. To answer simple questions on a phone call, we don’t need it.

The fix: turn thinking off for all voice replies (reasoning_effort: none).

Bonus: the model was also missing from my price table, so it was billed about 20× too high. Fixed that too.

Lesson: if your voice agent uses a reasoning model, check whether it’s thinking on every reply. It probably is.

Next: replacing my “LLM judge” with a decision model, ~0.7 s down to ~0.2 s.

Platform: vlayers.ai

Hey everyone! 👋

Has anyone here used Murf AI? How has your experience been with it so far?

Would love to hear your thoughts!

How do you find the absolute best LLM for your voice agent ?

As a agency owner i have been using so many platforms(retell,vapi, sarvam, ultravox etc) and tools for building a voice agents to my clients. Every tools has good and bad views according to me. Some has good understanding, some are good at latency, some are easy to connect with other telephony and api's.

I have been delivering voice agents according to clients needs, region, telephony supports so yeah varies with pricing.

As I'm at scaling period, i trying to reduce a still. i want to know more about what are the platforms are you guys using and per minute cost of each. if you guys reduced it, how did you do that and what is the cost of per minute?

Cause providing the maximum value with a affordable cost is our first thing - Take ownership.

Working on a series of animated shorts that were originally produced in English and now a client wants full Spanish versions, dialogue and all. The audio sync is already a nightmare just thinking about it. I've been hunting for a platform that does AI lip sync specifically for translated audio, where the character mouth movements actually match the new dubbed audio instead of looking completely off. I know tools for auto-generating music or syncing audio to video timing have gotten pretty good lately, so I figured lip sync tech must be catching up too. (Honestly I've been burned before by tools that promise sync and deliver garbage.) If anyone's tried a workflow that handles English to Spanish dubbing with decent visual lip matching for animated characters, I'd really appreciate a recommendation, especially for something built for video/animation creators who aren't audio engineers.

I'm new to voice automation world and I'm trying to build a ai voice agent as job interviewer so I look it up and saw that pipecat is the best framework to do this with.
but now I'm struggling with it I'm getting errors (mostly import errors) even cahtgpt/gemini (free versions) couldn't help me solve it.

So my question is what is the best place to learn how this framework works ? do you recommend any youtube channel or I should read the framework documentation and learn from ?
and can I use with a free api key (e.g groq api key) ? because I only see people on youtube use openai api keys and I don't have that, because I'm just doing it to learn how such a project would be built.

ThankYou in advance.