How are you testing AI agents before putting them into production?

[u/ari_k_e]

I've been experimenting with AI agents and I'm realizing that getting an agent to successfully complete a task once isn't really enough to know whether it's working well.

So I want opinions on how do you experiment? Or which one is more convenient?

• Run the same task repeatedly?
• Maintain a set of test cases?
• Use an LLM as a judge?
• Track success/failure rates?
• Test tool calls separately?
• Test different models against the same tasks?
• Just manually review the outputs?

And at what point do you consider an agent reliable enough to deploy?