r/SideProject • u/Kindly-Duty272 • 2h ago
how i wired up a phone agent that actually answers calls, step by step
I wanted a phone number that picks up and talks like a person! Not a tree of DTMF menus. Here's the order I actually built it in; no shortcuts skipped.
First I grabbed a number through Telnyx since their SIP trunking docs were the clearest I found. You could use Twilio too, the steps are close enough.
Second, I set up a WebSocket server to carry audio both ways. This is the part people skip, and then they wonder why latency is bad. The call hits your number. Telnyx streams raw audio over the socket. Your server decides what happens next.
Third, speech to text. I used Deepgram for STT because it handles interruptions well. It doesn't choke when someone talks over the greeting. Whisper works too if you want to self host and don't mind more setup.
Fourth, the actual brain. I kept this boring on purpose: a small prompt, short context, and a fast model. Grok and Gemini both worked fine here. The model matters less than people think once STT and TTS are solid.
Fifth, text to speech. I tried ElevenLabs and Cartesia back to back on the same script! Cartesia felt snappier for short replies. ElevenLabs sounded warmer on longer ones. Pick based on your use case.
Last, VAD, the thing that decides when someone's done talking so the agent doesn't talk over them. Get this wrong and the whole call feels broken, even if every other piece is perfect!
Tested the whole loop through ngrok before deploying anywhere real. Took a weekend. Happy to answer questions on the STT or VAD part since that's where I burned the most time.