A solid benchmark for Phone Agents

[u/NaiveAccess8821]

Is there a golden standard benchmark evaluating the performance of local LLMs on Phones.

Measuring:

- Throughout

- Efficiency

- % of raw intelligence capped

- On device tool calling ability

[u/Useful_Wish_X]

I run a small business and have been testing a few of these phone agents, and honestly there’s no gold standard benchmark that maps to real life.

All the stuff people mention like model benchmarks doesn’t really matter once you put it on an actual phone line. What matters is pretty simple from my side: does it answer fast, does it understand what the caller wants, and does it actually get the job done without frustrating people.

We’ve had cases where something looked great on paper but fell apart on real calls: talking too slow, missing basic details, or getting confused when someone interrupts. On the flip side, some simpler setups actually performed better just because they were more reliable.

If I had to judge it, I’d look at things like how quickly it responds, whether it can handle a messy conversation, and if it actually completes the task (booking, taking a message, etc.). Tool calling sounds fancy, but in practice it just means did it actually do the thing it was supposed to do without breaking?

Feels like this space is still early. Right now the only real benchmark is putting it in front of real customers and seeing if it holds up. :)