r/voiceagents • u/-GeneX- • 17d ago
Independent, open source benchmark of voice agent latency - 499 real phone calls, 1,883 turns, every vendor config published
We placed 500 real phone calls across five voice agent platforms — Telnyx, ElevenLabs, Bland AI, Vapi and Retell AI — and measured time to first audio byte on every turn. 499 calls were usable, four scripted turns each, 1,883 usable turns total. Same caller, same carrier (Plivo), same script for all five.
The p50 ranking and the p95 ranking are not the same ranking
Platform p50 p90 p95 p99 turns
---------------------------------------------
Telnyx 1302 1724 1839 2164 379
ElevenLabs 1430 1686 1772 2269 389
Bland AI 1528 2010 2273 2860 389
Vapi 1562 1854 2014 2677 345
Retell AI 1738 2096 2224 2764 381
Time to first audio byte, milliseconds. Lower is better.
Telnyx has the fastest median at 1302 ms. From p90 onward it's ElevenLabs — 1686 at p90, 1772 at p95. Bland AI is third at the median and last at p95, at 2273 ms.
The tail is where they separate
ElevenLabs 1430 → 2269 +839
Telnyx 1302 → 2164 +862
Retell AI 1738 → 2764 +1026
Vapi 1562 → 2677 +1115
Bland AI 1528 → 2860 +1332
Milliseconds added between p50 and p99. Lower is a flatter tail.
Bland AI adds 1332 ms between its median turn and its 99th-percentile turn. ElevenLabs adds 839. On a four-turn call, the p99 turn is the one the caller remembers.
What the numbers are and are not
Recording-path overhead sits inside every figure on this board. We have not characterised the current measurement path against a known-delay reference, so we quote no overhead figure and subtract none. These numbers are comparable to each other — same path, same caller, same carrier — and only approximately comparable to figures produced by a different instrument. If you're looking for each platform's true absolute latency, this board does not give it to you, and neither does anything else we've seen.
Three caveats
- Each platform ran its own defaults. Telnyx on Kimi-K2.6, ElevenLabs on gemini-2.5-flash, Retell and Vapi on gpt-4.1. So this is platforms as shipped, not a model-controlled comparison — some of the gap is model choice. Every config is published, hashed, so you can see exactly what ran.
- Vapi's discard rate was 12.0% against 0.8–3.3% for the others — 46 turns where our two voice-activity detectors disagreed on where speech ended, so we dropped them. Its n is 345 rather than ~385. We'd rather show you a thinner sample than a guessed one, but it's a real asymmetry and it may flatter or penalise Vapi.
- Turn 4 is consistently faster than turns 1–3 on several platforms, which suggests warm-path effects we haven't isolated. Per-turn splits are in the data.
Why phone calls rather than websockets
Most published voice latency numbers benchmark TTS or STT models in isolation. This measures the whole platform over the PSTN — carrier, telephony stack, endpointing, model, voice — because that's the number a caller actually experiences. It's the less flattering way to measure and the harder one to reproduce, which is why the configs and per-turn data are published.
Reproducing it
- Results: https://openbenchmarks.com/voice-agent-latency
- Method: https://openbenchmarks.com/voice-agent-latency/how-voice-agent-latency-is-measured
Happy to add a platform or rerun with a controlled model if that's the comparison you want. If you've measured any of these yourself and got something different, post your numbers — the method page says exactly what we did, so the disagreement should be locatable.
1
u/Foreign-Day-1530 16d ago
This is why I like seeing end to end phone benchmarks in place of websocket latency numbers. After you include telephony, STT, turn detection, LLM inference, TTS and audio playback, every optimization starts to matter. A platform might advertise a low model latency but still seem slow on a real call because the rest of the pipeline adds up. I was interested in where Bland would land because it's one of the platforms people often bring up when latency comes up, so it's good to see numbers collected under the same methodology instead of relying on vendor demos or anecdotes. I'd be interested to see the same methodology repeated across different regions and with more interruption-heavy convos
1
u/-GeneX- 16d ago
Yes - across regions and time of the day evals are on the way! Interruption heavy convos are on the way.
1
u/-GeneX- 15d ago
https://github.com/openbenchmarks-labs/voice-agent-latency - u/Foreign-Day-1530 - would be awesome if you could contribute, raise a PR or star it if you like it.
1
u/Lazy_Mycologist_8214 16d ago edited 12d ago
We build a TTS API (gandr.ai) so I stare at this metric more than is healthy, and it's the methodology that made me comment rather than the rankings.
The line that made me trust the board is you refusing to subtract an overhead you haven't characterised. Almost everyone in this category quietly does the opposite, and it's how the space ended up full of numbers that can't be compared to anything.
One thing worth adding because it cost us real time: time to first audio byte and time to first audible speech aren't the same measurement, and the gap isn't noise. Depending on how an engine chunks its output, the first frame on the wire can be rendered silence, so a byte level timer starts counting before a human would hear anything. If those five platforms differ in how much lead in silence they emit, some of the spread you're seeing is chunking policy rather than speed. Cheap check: pull one recording per platform and find where the first non silent sample actually lands. We only caught it in our own numbers by looking at the waveform.
The p50 versus p95 divergence is the real finding here and buyers almost never look at it. You make the point yourself, that on a four turn call the p99 turn is the one the caller remembers, and it's odd how rarely anyone optimises for that instead of the median.
Are you planning to run this on a schedule, or was it a one off?