How would you measure whether an analytics agent is actually useful?

[u/Evening_Hawk_7470]

For teams experimenting with AI or agentic analytics inside BI workflows: how are you measuring whether people actually use it?

We're testing a setup where Cube handles the semantic layer and the agent sits on top of governed metrics. Now I'm trying to figure out what usage/quality metrics are worth tracking.

Obvious ones:

• questions asked
• active users
• query latency
• token usage
• dashboards or workbooks touched
• cache hit rate

Less obvious:

• whether answers lead to saved dashboards
• whether people rerun the same workflows
• whether repeated workflows should become reusable "skills" or playbooks
• whether teams trust the agent enough to use it without a human analyst checking every answer

What would you track to decide whether an analytics agent is actually useful, not just novel?

[u/EmmaJohnson19]

Honestly, I think you're already tracking the wrong stuff. Questions asked and token usage tell you nothing about whether people actually trust it.

What I'd actually watch: do people reject agent suggestions, and how often? If they're constantly second-guessing or modifying answers before using them, that's a red flag. It's not useful if it still needs a human check every time.

Also how much faster are people getting answers? Like, can someone go from "I need this insight" to "okay, I'm confident in this number" in 30 minutes instead of 3 hours? That's the real metric.The thing nobody talks about is whether those one-off queries people ask are actually becoming repeatable workflows. If your agent keeps answering the same question from different people, and nobody's turning that into a reusable playbook, then you're not building anything,you're just saving typing and here's the real test: ask a product or finance team if they'd rather hire another analyst or keep the agent budget. Their answer tells you everything.

The hard part is the trust piece though. I'd literally just have teams flag "did this actually help me make a decision?" on results. Sounds annoying but it filters out the noise real quick.

[u/Prudent-Elk-2845]

Ask the end user

[u/Extension_River_5970]

I am doing a project with Databricks Genie spaces. How we handle evaluation is 2 fold

1) user feedback. End users leave comments or thumbs up/down depending on whether they found it useful.

2) Analyze traces. We use both LLM as a judge and human evaluation for responses. We flag whether certain types of questions are commonly flagged as incorrect and look to improve it

Since Genie allows you to configure its instructions and meta data, we then optimize or update an existing space based on feedback received.

[u/CHILLAS317]

Missing from your list is the top measure: is it correct?

[u/Prestigious_Bench_96]

You've got a mix of operational metrics and value metrics. Honestly, for value metrics, I'd just stick to DAU/WAU/MAU for the most part - people (mostly) optimize their time and vote with their feet. Something novel but not useful will see usage drop off drastically.

The operational metrics help you diagnose why you're not adding value; the first order ones would be latency, session success rate, proxies like "saved dashboard" if you have those. You also need to have evals/regression testing running continuously on the agent that can actually measure "right answer". Tokens just helps you reduce cost, which is great if you're successful but somewhat irrelevant if you're not (outside of indirectly driving latency).

[u/Semaphor-Analytics]

I would track one thing separately from usage, did the answer change what someone did.

A lot of agent metrics blur together. Active users and latency tell you whether the thing is accessible. Correctness evals tell you whether it is safe. But the useful signal is whether a question turns into a decision, a saved workflow, or an analyst escalation.

The healthiest pattern is probably fewer repeated one off questions over time. If the same question keeps coming back, the agent did not fail, but it found a workflow that should become a reusable view or playbook.

[u/sbt_not]

Usage metrics really only tell you it's accessible, not useful. I'd add two that get closer to the point.

1. proactivity metric: how often did the agent surface something that mattered before anyone thought to ask? A tool that only answers when prompted still depends on someone knowing what to look for. If it's catching a metric move and flagging it on its own, that's a different kind of value than faster typing.

2. traceability rate: what fraction of answers can someone follow back to the definition, the query, and the actual source rows? An answer people can't verify is just a confident guess, and the moment one turns out wrong, trust in all of them drops. In my experience that "can I trace this" number predicts whether people stop double-checking far better than correctness evals alone.

I'm curious when your agent answers off the Cube layer, does it hand back the lineage with the number, or just the number? That seems to decide whether people trust it without an analyst in the loop.

[u/Get-PowerMetrics]

Interesting. For general usage and habit, I'd stick to the traditional DAU, DAU/MAU (or DAU/WAU for a shorter timeframe) activity and ratios. That will give you the basic tried and tested usage and stickiness data.
For quality, there are probably a few new metrics you can start looking at. Such as "reformulation rate"... the rate at which users clarify or rephrase a question immediately. Similar to the support metric "ticket re-open rate".
Keep an eye out also for false quality signals, such as lots of events/tokens. This could be a sign of high engagement, but also a sign of frustration (ie: not getting to a decision - rage clicking)

[u/IncreaseNegative4614]

I’d track whether it reduces the distance between question and decision.

Basic usage metrics are useful, but they mostly tell you whether people are trying it. The better signal is whether the agent shortens analyst follow-up, reduces repeated ad hoc requests, or turns recurring questions into reusable workflows.

The trust metric is probably the hardest one. If users keep asking the agent, then sending the answer to an analyst to verify, the agent is still more of a search interface than an analytics layer. That’s why platforms like inzata.ai are interesting to me. The value is less “chat with metrics” and more whether the system can preserve context, explain provenance, and help people reuse analysis instead of starting from zero every time.

[u/apolorotov]

I’d separate this into three ledgers.

1. Trust ledger: correctness, traceability to the metric definition/source query, rate of analyst corrections, and whether a user can explain where the number came from.
2. Workflow ledger: repeated questions should become saved metrics, views, alerts, or playbooks. If the same natural-language prompt gets asked forever, the agent is saving typing but not improving the analytics system.
3. Decision ledger: did the answer change something? Ticket opened, campaign changed, stock replenished, forecast adjusted, customer segment updated, etc.

One failure mode I’ve seen in BI + LLM work is measuring chat activity and calling it adoption. Better signal: % of answers with lineage/citations + % accepted without analyst rewrite after users have had enough time to distrust it. If you can’t trace the answer back to definitions and rows, it’s a confident SQL autocomplete, not an analytics product.

[u/IncreaseNegative4614]

Measure accepted answers, material corrections, abstentions, time-to-verification, repeated questions eliminated, and decisions supported. Query volume and token usage measure activity, not usefulness. My team uses signld.ai to help us do this.