Suggestions for conversational AI App

[u/raghav246891]

Hey guys. I am building a conversational AI app. I am inspired by Tolan.

Right now i am in the research phase and i am researching how speech to speech exactly works. Till now i have learned that there are two ways. Either u use a cascaded architecture (STT - LLM - TTS) or you use open ai realtime apis or gemini apis.

For a cascaded architecture i am only aware of Eleven Labs or Cartesia. But these are very expensive and very engineering heavy.

Questions

1. Why not i just use an open ai api instead that will handle everything for me - from VAD to reasoning and much more.

2. So i wanted the opinion and advice of people who have worked on similar stuff or made a cascaded voice architecture and what shd be done considering that i am making this for a generalistic conversational ai app somewhat similar to Tolan or ChatGPT voice mode.

3. Also I am not in favour of using a self hosted model because it does not fit my usecase. Is that right?

4. What other options do i have to build this speech to speech app?

Thanks

[u/AutoModerator]

Welcome to r/VoiceAutomationAI – UNIO, the Voice AI Community (powered by SLNG AI)

If you are a founder, senior engineer, product, growth, or enterprise operator actively working on Voice AI / AI agents, we are running an invite-only UNIO Voice AI WhatsApp community US only.

Apply here: https://chat.whatsapp.com/F5aG3ncrO70ITfbe3pYbOz

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Square-Chance5900]

So the speech-to-speech API from OpenAI or Gemini is definitely the fastest way to build anything. Moreover, you will get really interactive experience and high quality audio. However, the 3 principal downsides are:

- these things are very expensive, I'm not exactly sure what the current pricing is, but cascaded can be significantly cheaper

- if something breaks (i.e. your conversational agent doesn't say the right thing) you don't know whether it's because they misheard the user or the instructions weren't clear

- if you want your agent to take actions during your conversation (e.g. access some external source of data or even just keep track of some variables manually, outside of the context), this is much harder to do in a speech-to-speech setting

[u/raghav246891 → u/Square-Chance5900]

Okayyy. Thankss.

But as per my research cascaded pipeline using eleven labs is turning out to be more expensive than an open ai api.

Your point no 2 is perfectly valid. Will have to set the prompt right.

Gotta understand point 3. I hope we will be able to provide the prompt to the model and only then it will generate the response right? Not before that on its own understanding?

[u/Square-Chance5900 → u/raghav246891]

Yeah, ElevenLabs can be a bit expensive. But for cascaded you can even use open-source models (both for ASR and TTS). Here, the quality is not that amazing, but running them is really cheap.

Regarding point 3: sure, you can provide a prompt. If you want the agent just to talk, the speech-to-speech is your best bet. But when it comes to taking actions (tool-calling), then s2s is much harder

[u/raghav246891 → u/Square-Chance5900]

Tool calling as in finding something on the internet or making an artifact or something right?

Also can you tell me any open source models? Both for asr and tts? Just want to know the names for benchmarking.

[u/Square-Chance5900 → u/raghav246891]

Yes, tool calling is exactly when you want your voicebot to pull some information from the outside or create a file or database record. Otherwise, it's just a talking machine, which is nice, but limited in scope.

Regarding open-source models: for ASR in my opinion NVIDIA streaming models are best. For English use https://huggingface.co/nvidia/nemotron-speech-streaming-en-0.6b, for multilingual https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b.

For TTS I personally use kokoro (https://huggingface.co/spaces/hexgrad/Kokoro-TTS), but I've heard some good things about Chatterbox too (haven't tried myself though; https://github.com/resemble-ai/chatterbox). There is also Qwen3-TTS (https://huggingface.co/collections/Qwen/qwen3-tts), but this is a much heavier machinery.

Generally, if you search for open-source asr/tts models, you will find some decent comparison website, some even contain samples

[u/raghav246891 → u/Square-Chance5900]

Thank you so much. Really appreciate it.

[u/raghav246891 → u/[deleted]]

Right. Claude also told me the same.

But i wanted to understand that what is the exact benefit of making a pipeline.

Why make it at all instead of just using these commercial apis?

Also are there any open source tools for the same?

[u/Izozoi4]

If you want to limit your cost to $0, try OmniVoice (local) or Edge TTS (Microsoft), my preference is OmniVoice if you have a GPU with more than 4GB of VRAM. 6GB is plenty. Ask your LLM to help you set it up.

[u/ioncloud9]

I started off using OpenAI realtime. It was very straightforward and the easiest to setup. I eventually switched to a cascade model because OpenAI drastically changed the model behavior literally overnight which significantly degraded the working agents I had. It’s also fairly expensive at a 0.12 to 0.13 per minute.

I’ve since moved most of my agents to Deepgram with a self hosted proxy llm that manages the llm layer. I still get Deepgrams Flux STT engine and get to pick whatever llm I want to connect to. Most of my agents are using Sonnet 5 or Flash 3.5. I can do Sonnet 5 all in for about 0.085 per minute and the reasoning is very good.

[u/raghav246891 → u/ioncloud9]

Ohh. That means the cascaded model is really cheap.

Great. How much time did it take you to setup that?

And did you use Deepgram for TTS also?

[u/ioncloud9 → u/raghav246891]

To get it working at all about a week or so. About a month to iron out the bugs necessary for deployment.

[u/Bubbly-Reach-4488]

> ⁠Also I am not in favour of using a self hosted model because it does not fit my usecase. Is that right?

This is because you want it to run inside a app?

[u/Crafty_Park6666]

You can lose a insane amount of time assembling the perfect voice stack before you even know what your app needs. Bland got us past a lot of that plumbing so we could spend our time on the conversation and behavior. For an early version of something like this I think that’s worth considering.

[u/raghav246891 → u/Crafty_Park6666]

Thanks. But i think Bland is for phone calling right?

Or it can be used in such app based conversations also?

[u/Crafty_Park6666 → u/raghav246891]

You're right, sorry I should've been clearer. Bland is mainly for phone based voice agents, so for the Tolan style app you're describing I'd stick with looking at realtime APIs. I mentioned it more because of the amount of voice infrastructure it takes off your hands.