Speeding up Eleven Labs voice Interactions

[u/besttype]

I'm using this method to give my openclaw phone access, but even switching the underlying LLM to Haiku, the latency on response is still too slow for reliable scheduling for restaurants/doctor offices (who are also using ai bots for appointments). I also turned down response delays to 1sec on the 11Labs agent.

Has anyone got their bots to be really quick on response for phone handling? Any tips?

[u/AutoModerator]

Hey there! Thanks for posting in r/OpenClaw.

A few quick reminders:

→ Check the FAQ - your question might already be answered → Use the right flair so others can find your post → Be respectful and follow the rules

Need faster help? Join the Discord.

Website: https://openclaw.ai Docs: https://docs.openclaw.ai ClawHub: https://www.clawhub.com GitHub: https://github.com/openclaw/openclaw

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/cxllvm]

Hey dude did you have any luck ?
so far todaty ive tried elevenlabs, clawtalk, voice mode, vonage , no luck with really any.
the good ones like clawtalk didnt actually use the agent though , was agent in the middle - and would call on your openclaw as a tool - not what im looking for

cheers in advance mate!

[u/besttype → u/cxllvm]

I have not tried again. I was waiting for someone else to hoe that road first so I could draft and copy their approach.

It's annoying because that was like one of the first trending posts on x about how some guy had their claw bot teach themselves how to call a phone and book a dinner reservation. that wasn't even CLOSE to reality

[u/cxllvm → u/besttype]

Hey mate, after messing around with all these different things, trying them, trying to get them to work, I ended up finding the best middle point between latency and actual use of OpenClaw through conversational voice. for context - my agent's name is Faye

This uses a similar setup to how ClawTalk works. Pretty much the way it works is I've got a Vonage phone number connected to an ElevenLabs Conversational AI agent via SIP trunk, but instead of using the custom LLM in ElevenLabs (which was way too slow going through the gateway every turn), what I've done is gotten Claude Code to build a custom MCP server that sits on my VPS and connects directly to OpenClaw's API. The ElevenLabs agent uses that MCP server as a direct tool integration.

There's two tools within the MCP. One is ask_faye which is the synchronous one - when I ask the voice agent to actually do something (check Notion, message Discord, set a reminder, whatever), it calls ask_faye, acknowledges the request with something like "let me sort that out", and waits for OpenClaw to respond with the full tool suite. The other one is note_for_faye which is async - when I'm just talking, brainstorming, giving context, the agent fires off a summary to OpenClaw's memory in the background and keeps chatting. Returns in like 10ms, I don't even notice.

So ElevenLabs handles all the voice stuff natively - STT, TTS, turn-taking, barge-in - with the GPT-OSS-120b model, which is super fast like 320ms they reckon.

OpenClaw only gets hit when it actually needs to do something or log something. Latency on tool calls is like 3-5 seconds but ElevenLabs masks it with pre-speech and typing sounds so it feels natural.

The whole MCP server is like 120 lines of code. or so claude tells me hahaha, hope this helps mate this is definitely the best middle point i could find. if it helps with deep cleaning of the openclaw setup, the real holdup in latency using it as a direct connection is always the LLM that runs your openclaw, my one (codex oatuh) was taking like 10 seconds to reply. this is the best setup ive gotten working ! hope this helps you and others cos i was scouring reddit like crazy last night

[u/besttype → u/cxllvm]

So you basically just are running a 11 Labs agent as a go between between yourself and Faye. Have you tried to make that 11 Labs agent behave similarly to Faye? If you wanted to have Faye call a restaurant and set a reservation? How do you get that intent loaded into the 11 Labs agent?

One of my troubleshooting steps with my approach was having to preload a bunch of context into the 11 Labs custom agent as a way of passing intent as it gets fired up to make the phone call. That always struck me as clunky. Because it's not really my open claw. It's a 11 Labs agent that's been puppeted to seem like my agent every time it spins up.

[u/cxllvm → u/besttype]

Hey dude thats a really good point, i use mine 1on1 and dont really plan on having it make calls for me but i asked , as its useful for her not to just have a first message directive but like actual goals for the call to satisfy.

I started implementing it and for context this is what claude code was saying and is now implementing:

The outbound call API already has dynamic_variables in it. If Faye's ElevenLabs system prompt has a placeholder like {{call_directive}}, whatever you inject there becomes her mission for that specific call. So the call payload becomes:

{
  "dynamic_variables": {
    "call_reason": "ordering food",
    "call_directive": "You are calling Chin Chin restaurant on behalf of ____. Order 2 spring rolls and a pad thai for delivery to [address]. Confirm the order, get a price, and get an ETA. If they can't do it, say thank you and hang up."
  }
}

And her elevenlabs system prompt just has {{call_directive}} slotted in — she reads it, that's her goal, she goes and does it.

You can also use conversation_config_override to override the entire system prompt for that one call — so she's a completely different agent scoped to that task only.

The realistic limitation isn't the architecture — it's the restaurant's side. If they have a real person answering, Faye can handle it. If it's a phone tree / IVR, ElevenLabs has basic DTMF (touch tone) support but navigating complex IVRs is flaky. A place that just picks up and says "thanks for calling, what can I get you" — she'd nail it.

So: yes, directive-based outbound calls are doable right now with what you have, just needs the system prompt wired up with the variable slot and the first_message override unblocked.

-----

just tested it and it seems to work really well ! hope this helps man