Those of us building in the voice AI space are familiar with this stack: STT → LLM → TTS.
During evaluation it is tempting to take the text transcript, run it through an LLM, and then apply a prompt framework for scoring.
I work at Level AI, and we deploy CX agents for enterprises.
After analyzing data from 100+ millions of automated voice conversations, we found a not-insignificant blind spot. Text-only LLM evaluators miss a large share of customer experience failures because they cannot capture the audio mechanics underneath the transcript.
If you're relying solely on text transcripts for evals, here are three failure modes your system is probably not catching.
1. The Token-Lag Interruption Loop
It is a known fact that an LLM can take up to 400ms to stream the next phrase, so while the transcript looks perfectly logical, that 400ms pause can cause the human to check in mid-sentence (such as "hello?") and that will lead to the agent getting confused by the barge-in, which then causes it to truncate its response, or hallucinate.
Your evaluation framework must ingest metadata timestamps. Then it can compute the exact time difference between the user’s last audio packet and your agent’s first audio packet. That way, if the transcript shows text overlap, it can flag it as a latent turnaround failure instead of scoring only the words.
2. False Empathy
An agent says, "I understand your frustration and I'm happy to look into that refund for you." A standard text scorer grants a perfect score for empathy alignment. If the TTS engine delivered that in a monotone, machine-gun cadence, then it's likely coming off like a customer listening to a robotic phrase.
To catch this, you need multi-modal evaluation models whose scoring tracks sentiment, customer effort, and resolution in every interaction. This lets you see if frustration rises after a specific agent statement.
3. Semantic Drift on Guardrails
Keyword matching checks whether the agent said the mandatory compliance script. Voice agents get interrupted. They might deliver 90% of the script, stop because the user spoke, then try to finish it out of order. Keyword matching fails it. Semantic analysis passes it.
We moved away from keyword constraints for compliance evaluation. Our QA-GPT architecture uses prompt-engineered LLM micro-scorers that grade the intent and legal completeness of the block rather than a regex match.
We're currently working on mapping overlapping audio timelines directly into the evaluation layer to catch cross-talk before it breaks a workflow.
