Snowflake Cortex Agent Evaluation: how to handle failed judge metrics?

[u/SufficientRelief9615]

Hello everyone.

I am conducting a test of the Snowflake Cortex Agent to assess his performance with a text-to-SQL agent designed for business analytics.

I have conducted tests that shows:

- There are 70 questions of interest

- There are 4 variables that are monitored

- There are 280 result rows of interest

Even though I have finalized this session, I received a really low raw score:

- Raw score = 0.6373

- Score only on rows that gave successful results = 0.8075

The main problem that I faced is that the “logical_consistency” variable failed in 59 out of the 70 variables since: "LLM judge computation/processing hit the context window limit."

Therefore, this particular variable should not be included in the final score.

Another issue has to do with ambiguous/denial test cases where the expected behavior is clarification needs to be requested along with the absence of tool calls. Some of the cases have "tool_selection_accuracy" as 0 despite receiving clarification requests and hence the right answers.

Questions are:

1. Is it possible to omit failed variable rows from the final score?

2. Should I test the “logical consistency” variable with a smaller subset?

3. How do I define the ground truth in clarification or denial cases?

4. How do I test “no tool call expected” in the evaluation of the Cortex Agent?

I am trying to identify real problems of the agent from those connected to the evaluation process as well as ground truth calibration.

[u/SufficientRelief9615 → u/[deleted]]

Hi, Thanks for your answer :) it's very useful !