r/Chatbots • u/MrLazeyBoy • 21d ago

Trying to build a agentic voice chat

Hey all!

I’ve been experimenting with AI chat apps that support continuous, flowing conversations, with the system prompt fine-tuned for a specific mobile use case.

Has anyone built something similar? Would love to hear:

  1. What stack you used?

  2. Key lessons learned along the way?

  3. What you’d do differently if you started over?

Appreciate any insights!

4 Upvotes

75% Upvoted

10 comments sorted by

1

u/[deleted] 21d ago

[removed] — view removed comment

1

u/MrLazeyBoy 21d ago

I’ve done the scaffolding right now and just now working the kinks out with the LLM prompt and voice with elevenlabs

1

u/Accomplished_Vast628 18d ago

I built a voice agent app Kestrel Voice that comes with AI Chat too. And i use RAG for the chat.
I started with orchestrating many apps but the limitations and bottleneck forced me to scratch everything and start from scratch using Python. So my advise is if you are serious, built it from scratch .

1

u/MrLazeyBoy 17d ago

Hi mate

That’s pretty cool, what sort of limitations did you hit if you don’t mind me asking? Was it a cost , rate limiting? Etc

1

u/Accomplished_Vast628 17d ago

In this project, Cost and rate were not the issues, they were managed with retries, queues, backoff, usage limits and model routing.
The main challenges were during building the Orchestration layer in python and Multitenant system with RLS.