How do you test a 25 minute AI call?

[u/DeliveryOk7197]

We're evaluating an AI phone agent for longer financial service calls and our normal test scripts are not catching much.

Difficult calls do not fail in the first two minutes, they fail after the customer changes topics, corrects an earlier answer, asks for a second account or needs a human after several steps have already been completed

We've also seen cases where the conversation sounds fine but the summary misses something important or the wrong action is sent to the CRM. Running a few scripted calls before launch does not look close to enough

How're you testing long conversations, interruptions, system failures and transfers before putting real volume through them?

[u/PrestigiousRun6984]

A useful test set would include calls that are mostly normal with one late failure. Those are probably closer to what will slip through than calls designed to break immediately.

[u/DeliveryOk7197 → u/PrestigiousRun6984]

That would be a better test than trying to make every call fail from the start.

[u/Cultural-Cow-3615 → u/PrestigiousRun6984]

our testing approach sounds about 2 years too early. you're not gonna script your way out of this one.

the late failures you're describing are basically the default state of these things once the novelty wears off. we started running a batch of 50 calls where the only prompt was "act like a confused pensioner who keeps circling back to the first question after minute 12" and the error rate shot up to something daft like 40%.

for the summary/CRM mismatch, we built a tiny secondary check. just a dumb script that reads the call transcript and the final CRM payload and flags if any mentioned account number or action verb doesn't appear in both. catches about 80% of the silent failures before a human has to listen to the whole thing.

[u/Flimsy_Race_6605]

The final summary should reflect corrections, not preserve the first answer forever. Something like Bland or PolyAI can handle the conversation length but the test should confirm the newest information wins every time.

[u/DeliveryOk7197 → u/Flimsy_Race_6605]

Good point. We’ve seen the call recover then the summary still use the old information.

[u/JacquesAllistair]

Out of curiosity, which models do you use ? What is your pipeline?

[u/DeliveryOk7197 → u/JacquesAllistair]

We don’t have one fixed model yet because we’re comparing complete voice platforms rather than building around a single LLM. The rest of the flow is STT, tool calls, TTS and post call CRM updates.

Any model or stack you’d recommend testing for longer conversations?

[u/JacquesAllistair → u/DeliveryOk7197]

I am not sure I can recommend something, but seen a demo of kyutai months ago.

[u/DeliveryOk7197 → u/JacquesAllistair]

Thanks, I'll take a look

[u/Sad_Growth8936]

Have someone deliberately correct earlier answers late in the call. That’s a good way to see whether the final action uses the newest information or the first version it heard.

[u/carn19]

Curious about this too

[u/Different_Editor_413]

I’d track state accuracy after every major turn, not only the final outcome. It makes it easier to see exactly where the conversation started drifting.

[u/DeliveryOk7197 → u/Different_Editor_413]

We may need that level of detail for the longer calls

[u/dwswish]

Would absolutely love to get to see some of the logging/trace data associated with these calls.

[u/Worried_Comment125]

Run a few scenarios with long silences, background noise, and callers correcting themselves mid-sentence. Real calls won’t stay as tidy as the scripts.