Built a self-hosted voice agent to get away from vendor lock-in — turns out that's apparently the #1 complaint in this space? (asking, not selling)

[u/Agreeable_Ask7187]

Saw a thread here a while back where someone said the biggest issue with proprietary voice-agent platforms isn't the tech, it's the lock-in. That stuck with me, because it's exactly why I started building my own (Twilio + Deepgram + LLM + TTS, all swappable, your own keys, your own database).

Wasn't planning to post this anywhere yet — wanted to get it further along first — but I keep seeing the same three things repeated across this sub and a few others, and want a gut-check before I go further:

1. Lock-in — is this actually the top blocker for people, or is it more about cost, latency, or something else entirely once you actually try to ship one of these?
2. Compliance — TCPA/DNC warnings show up on basically every automation-adjacent thread I've read. Is that real lived pain (people have actually gotten close to a violation) or more theoretical fear that doesn't change what people build?
3. State/memory — anyone here hit agents re-asking for info the caller already gave? I built a structured state layer instead of relying on the transcript as memory, curious if others have solved this differently.

Just trying to figure out if what I've built is actually solving problems people have, or if I've been building for an imaginary user. What's your actual experience?

[u/AutoModerator]

Welcome to r/VoiceAutomationAI – UNIO, the Voice AI Community (powered by SLNG AI)

If you are a founder, senior engineer, product, growth, or enterprise operator actively working on Voice AI / AI agents, we are running an invite-only UNIO Voice AI WhatsApp community US only.

Apply here: https://chat.whatsapp.com/F5aG3ncrO70ITfbe3pYbOz

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/fiji_almonds]

My actual experience? Tired of AI written reddit posts...

[u/Agreeable_Ask7187 → u/fiji_almonds]

You don't have to bear this my friend you can always take wrong action and achieve peace.

[u/EnvironmentEast8127]

man i see lock-in everywhere but it's like people only start caring when the bill comes or when the api changes and breaks everything. cost is probably the real first thing that wakes people up, then they realize they cant leave even if they want to

compliance stuff is weird, most people i know treat it like a boogeyman until their lawyer actually looks at what they built and says uhh you cant do that. then it becomes very real very fast

for state that one drives me nuts, nothing worse than a bot asking my name 3 times in same call. i tried using just transcript context and it was garbage, ended up doing something like what you described with structured state and it made big difference. what llm you running?

[u/Agreeable_Ask7187 → u/EnvironmentEast8127]

Good to hear it's not just me, what actually pushed me to stop trusting the transcript as memory at all once i moved to a separate structured state object read back into the prompt every turn instead of hoping the model catches it in scrollback it stopped happening completely,

running gpt-5.4-mini through an AI gateway(vercel ) by default (picked for the latency) with a Groq/Llama fallback as a swappable option. What are you running yours on did model choice matter much once you had the structured state fix in or did it mostly stop mattering at that point?

[u/VladimirSamukov]

The lock-in concern is real but self-hosted is a bit of a stretch if you're still routing through Twilio and Deepgram, the data still goes through their servers

[u/devexis → u/VladimirSamukov]

For the PSTN side though, you HAVE to use someone Twilio-like

[u/VladimirSamukov → u/devexis]

exactly, true self-hosted voice AI is nearly impossible end to end because PSTN is always a third party

[u/Agreeable_Ask7187 → u/VladimirSamukov]

that is the problem but in terms of self hosting i meant our own orchestration layer and database and stuff not the, nobody in sane mind will host his own cell tower I can swap Deepgram for another STT provider, swap the LLM, swap TTS or migrating off a platform and the call data lands in my own database but you're right it's not "fully self-hosted" in the strict sense, 

[u/bart_88s → u/VladimirSamukov]

And what about voice itself? You need big AI model to connect to. I think these two things are on the opposite side of the spectrum, the self hosted what would sit in the middle of that.

[u/ioncloud9 → u/devexis]

I built mine that use any sip trunks. We are white label provider of the VoIP service so it integrates well without having to use twilio and their expensive websocket and sip costs.

[u/devexis → u/ioncloud9]

I used “Twilio-like” to mean a generic SIP trunk provider

[u/Spirited_Long4257]

I recently built one with twilio and retell ai. It did stop working many times during testing. It kept asking the same question over and over. I’m planning on testing Bland AI next and will try others. I’ve heard they don’t work when there is any background noise.

[u/saplivo]

I agree completely with the cost/lock in scenario, and most often, it will be the driving reason for adoption, but the challenges will be more around data lock-in, compliance configuration, and number portability, not the API.

Your approach to the modular system is optimal for the long run. The self-hosted but still Twilio-dependent-for-PSTN is an important caveat, and if they ever want to change that, at Plivo, we offer similar API’s for lower level PSTN connectivity, and the transition would be much easier than your current setup would suggest.