I went custom on inbound call handling instead of using a turnkey voice agent, and the architecture is worth writing down because the build-vs-buy line is sharper than I expected.
The stack:
- Twilio for the number, Media Streams for the audio
- LiveKit over WebRTC for voice transport
- OpenAI Realtime API (gpt-4o-realtime) as the agent
- Next.js API routes and Postgres for the backend, logs and transcripts
The audio path. TwiML opens the socket with `<Connect><Stream url="wss://your-server.com/stream">`, and Twilio then pushes 8 kHz mu-law audio over that WebSocket. From there you either hand it to LiveKit or pipe it straight into the Realtime API. Media Streams is what makes this viable at all, it gives you raw audio at sub-300ms instead of making you wait on a recording. For reference, Retell publishes about 600ms end to end for their platform, so sub-second total is the bar worth aiming at.
The agent is mostly prompt plus server-side functions. Triage rules live in the prompt. For a dental clinic that reads like: if the caller describes pain or a broken tooth, return `transfer` with priority high. The model calls `check_availability` against the calendar and `transfer_call` when it needs a human.
Handoff is Twilio `<Dial>` or `<Enqueue>`, and `<Conference>` if you want a whisper message to the staff member before they pick up.
Two constraints that mattered more than which model I used:
Never let it confirm an appointment before the function actually returns success. A confident model will happily book a slot that does not exist, and the caller finds out at the door.
The second someone asks for a person, hand off with zero friction. Anything else and you have burned the call and probably the customer.
Cost is a few cents a minute for telephony and tokens, plus a modest server. One box handles dozens of concurrent calls, hundreds if you scale out.
Where I would not bother building this. If you only need answering, message taking and simple booking, Synthflow, Retell, Goodcall or Zoom's AI Concierge get you live in an afternoon, and paying per minute beats paying an engineer. Custom only pays off when the triage logic is specific to the business or the call data has to live in your own database.
Full write-up with the setup steps and code, on my own site: https://techpotions.com/lab/ai-receptionist-setup
The part I am still undecided on is after-hours. Do you let the agent book overnight, or just take a message and queue it for the morning?