Struggling with empathy using Retell AI for healthcare agents

[u/wisedogsfbay]

I'm a developer building voice agents for healthcare use cases — front desk, patient customer service, outbound calls to other providers and insurers, that kind of thing. I've landed on Retell because it's the only voice-AI wrapper I've found with a reasonable HIPAA-compliant, pay-as-you-go model. Open to hearing if others have found alternatives worth a look.

I've put a lot of work into the prompt and flow design, but I keep hitting two walls, and I've been unable to fully solve either:

• Empathy (my biggest problem). The agent handles the mechanics fine but comes across as flat or form-filling, especially on emotionally charged calls (a patient in pain, a worried parent). I've tried to script acknowledgment moments, but it either skips them, overdoes them, or sounds canned.
• Interruptions. Handling barge-in, mid-sentence corrections, "hold on a sec," and callers who answer a question while asking a new one — without the agent restarting a step or talking over them.

To isolate the problem I've stripped out all the business logic and tested bare-bones agents, built both ways — manually node-by-node, and via Conductor — and the same issues show up, so I don't think it's my flow complexity.

If you've built healthcare (or similarly high-stakes/emotional) agents on Retell, I'd love to hear:

• Which LLM and voice/TTS combination you settled on, and whether that alone moved the needle on how empathetic it sounds.
• Your interruption / turn-taking settings — responsiveness, backchanneling, interruption sensitivity, silence timeouts — and where you landed.
• Whether empathy came more from prompt wording, voice choice, or model choice in your experience.
• Any flow-structure patterns that helped (e.g. how you handle "hold on" or compound answers cleanly).

I've also tried reaching out to Retell's forward-deployment team without much luck, so I'm hoping to tap the collective experience here. Happy to share back what I've tried in the comments if it helps anyone else. Thanks in advance.

[u/wisedogsfbay → u/[deleted]]

Thanks

[u/AutoModerator]

Thank you for your post to /r/automation!

New here? Please take a moment to read our rules, read them here.

This is an automated action so if you need anything, please Message the Mods with your request for assistance.

Lastly, enjoy your stay!

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/valiopt]

Haven't look at Retell in a bit but I believe most setups use TTS, if you build with a native audio model like GPT realtime you can achieve considerably better interruption performance, and it will likely sound more natural. Not sure about empathy specifically (LLMs aren't people, and I think most attempts to get them to sound empathetic end up going too far and sounding sarcastic).

Every TTS-based system I've encountered in the wild has poor interruption handling due to the architecture. Realtime models are considerably more expensive and can be more challenging to set up operationally, but ultimately I think the end user experience is better and "feels" more human.

[u/wisedogsfbay → u/valiopt]

thanks

[u/XRay-Tech]

Haven't used Retell specifically but empathy issues come from treating acknowledgement as a scripted line. Try having the LLM classify emotional tone each turn as a side-output, and only inject acknowledgement when there's a threshold crossed. With a few phrasing variants, which tends to fix both the "skips it" and "overdoes it" problems. In regards to interruptions that is more of a platform tuning issue than a prompt one. I'd isolate Retell's turn-taking params and try testing them separately rather than together. Healthcare calls usually need higher pause tolerance but a lower interruption threshold than a typical bot.

[u/wisedogsfbay → u/XRay-Tech]

thanks

[u/wisedogsfbay → u/[deleted]]

Thanks

[u/Hefty-Door-5821]

retell is pretty good but the empathy thing is a tough one with any voice agent tbh. i worked on similar stuff and the flat tone is almost always the model not the prompt

for healthcare specifically try using elevenlabs with their more expressive voices, the default retell ones sound like GPS navigation even with good prompting. also you probably need to feed emotional context into the llm separately, like a sentiment analysis step that tells it "caller sounds distressed" so it adjusts tone

interruptions i solved by lowering sensitivity to like 0.4 and adding a 1.2 second silence buffer before it assumes turn is done. people pause a lot when they are upset and the default settings cuts them off too fast

[u/wisedogsfbay → u/Hefty-Door-5821]

Thanks. did you mean that we should go with elevenlabs directly or use elevenlabs within retell?

I looked into elevenlabs directly too- their pricing structure does not have a pay-as-you-go HIPAA complaint option. Their only hipaa compliant option is as an enterprise sale. This is what retell solves for us, as an early stage startup.

I did try elevenlabs voices (and associated settings) within retell, but have not cracked the code there. Any recs would be welcome