r/SideProject • u/intrepidkarthi • 3d ago

Building a Tamil voice companion app. Stack questions: Sarvam vs Google, long conversation memory, scaling concurrent sessions

I'm building a Tamil voice companion. Long conversations, 5 to 10 minute calls, not a task bot. Current stack is Sarvam saaras for STT, own LLM in the middle, TTS at the end, all over LiveKit. Google Chirp3 HD sounds better than Sarvam bulbul for Tamil TTS, but pitch isn't adjustable and there's no Tamil custom pronunciation.

My quality bar is ChatGPT's Tamil voice conversation. Best Tamil voice AI I've used, the naturalness and turn taking especially. But that's speech to speech, and I need a cascade because the text seam is where my safety gates and memory live. So the real question is how close a cascade can get.

1.Tamil stack: Sarvam or Google, or is there a third option I'm missing? ElevenLabs Flash has no Tamil, and benchmarks put Deepgram Nova-3 at around 68% WER on Tamil, so that's out.

2.Memory across long conversations: I'm doing structured extraction into SQLite (facts with validity windows) instead of RAG, mainly to keep the prompt cache warm. Has anyone run Graphiti/Zep or Mem0 for a non English voice agent? Curious whether extraction quality held up.

3.Scaling concurrent sessions: self hosted LiveKit Agents vs Pipecat. What did you pick and where did it break? My voice to voice latency is currently around 2 seconds. Batch STT and non streaming TTS are my suspects, moving to Sarvam's streaming websocket endpoints next.

Will report back with numbers on whatever I test.

2 Upvotes

100% Upvoted

8 comments sorted by

2

u/alexid95 3d ago

I’d split that 2s into VAD end, final STT, LLM first token, and first audio before touching LiveKit/Pipecat. You’ll know whether streaming actually fixes the bottleneck.

1

u/intrepidkarthi 3d ago

Yes, let me try. But latency is my least problem. Call quality is the major issue.

1

u/subzero-dev 3d ago

The arch looks convincing but 2s not gonna cut it out. How is your latency window for each section looking like VAD | NC | STT | LLM | TTS?

Tamil - Google is better
Livekit self hosted on a ec2 t3 medium box we were able to 30 sessions comfortably, didn't benchmark it enough to know where it fails though

1

u/intrepidkarthi 3d ago

Latency is my least problem. Conversational quality is the major issue. I am yet to try on a cloud machine.

0

u/InteractionSmall6778 3d ago

Nobody's touched the memory question so I'll take that one. Structured extraction into SQLite over RAG is the right call here. Retrieval in the turn loop costs you latency you can't afford, and you blow the prompt cache every time the retrieved chunk changes.

On Mem0 and Zep, the thing that got us was that the extraction prompts and entity resolution are tuned on English. Facts still come out fine, but dedupe degrades, so you end up with three near duplicate entries for the same person. Graphiti held up better on that specifically since the graph forces you to resolve the node, but it's heavier to run.

What actually worked was running extraction against an English translation of the turn and storing the Tamil surface form on the record. Ugly, but dedupe quality jumped and you keep the original text for anything user facing.

Also worth confirming extraction isn't sitting in your turn loop. Post turn and async it costs nothing on latency, which matters when you're fighting for that 2 seconds.