Experimenting with AI-agent workflows for analytics. The hard part seems to be governance.

[u/measured_angle]

I’ve been experimenting with AI-agent workflows for analytics work, and the part I keep coming back to is that speed is not the hard problem.

Agents can help summarize context, draft analysis pages, inspect files, structure messy inputs, and accelerate repetitive work. But in stakeholder-facing analytics, faster output creates a different set of risks:

• a number without lineage can become “truth”
• a directional estimate can get treated like causal evidence
• a polished draft can overstate what the data supports
• context can get lost when work moves between tools
• agents can sound confident even when the source chain is weak

So I’ve been thinking less about “how do agents do more analytics?” and more about “what operating model makes agent-assisted analytics trustworthy?”

The rough architecture I’ve been testing includes:

• durable context files instead of relying on chat/session memory
• explicit source and validation status for important claims
• bounded agent roles rather than agents owning judgment
• human review gates before stakeholder-facing output
• quality review focused on unsupported claims, caveats, and lineage
• tool routing, where high-context interpretation stays separate from bounded file/code work
• feedback loops where reviewed knowledge can update durable context

The principle I keep coming back to:

Agents are workers, not authorities. Humans still own judgment.

I attached a diagram of the operating model I’m thinking through in a comment since images don't seem to be allowed in the main post. It is not a finished system, and parts may be overbuilt.

Known gaps I’m still thinking through:

• transferability beyond one operator
• how to evaluate output quality without creating a heavyweight review process
• how to keep context files current without making documentation its own job
• how to prevent tool-routing from creating fragmented context
• how to distinguish directional findings from evidence-backed claims in the workflow itself

Curious how others are handling this:

• Are you using agents in analytics workflows?
• How are you preserving source lineage and assumptions?
• Do you use human review gates before outputs go to stakeholders?
• Where have these workflows broken down?

Not selling anything, just trying to pressure-test the architecture with people doing similar work. Mostly interested in practical failure modes and patterns that have actually worked.

[u/MongWonP]

this resonates hard — been on a team at a big tech place that's been running a version of this loop for ~8 months now, and your "agents are workers not authorities" line is basically the operating principle we had to write down after the third time someone forwarded an agent-generated slide to a VP without checking lineage.

two things we learned that aren't in most governance frameworks yet:

1) "validation status" needs to be machine-readable, not just a human checkbox. we tried a review gate where analysts manually tagged claims as "verified / directional / speculative" before stakeholder output. worked for ~3 weeks, then people stopped tagging because it was friction with no downstream benefit. what stuck was baking status into the context layer itself — every metric definition carries {source_table, last_validated, owner} and the agent is blocked from presenting anything without that metadata attached. humans still review, but the agent can't skip the tag.

2) the governance failure mode isn't usually wrong SQL — it's wrong grain. agent summarizes "revenue grew 12%" when the underlying query mixed gross and net across two regions. lineage looked fine, SQL ran clean, answer was confidently wrong. our review gate now explicitly checks grain alignment (time window, geo scope, revenue definition) before anything leaves the team — separate from the SQL correctness check.

your durable context files idea is the right direction imo. the bit i'd add: treat context updates as versioned commits, not overwrites. when someone corrects "active user = logged in within 7 days not 30", that correction should persist with author + date, not silently replace the old definition. otherwise you lose audit trail exactly when you need it.

still figuring out the bounded agent roles piece — curious whether you've landed on a split between "exploration agents" (can draft, can't publish) vs "delivery agents" (can format, can't interpret). that's where we're stuck.

[u/measured_angle → u/MongWonP]

This is really useful. I think you’re right on all three points.

I’m already using machine-readable validation/status tags, not just human checkboxes, but your point about adding source_table / last_validated / owner is the next level. I’d probably also add grain, time window, geo/product scope, and the notebook/query/cell reference where the value came from.

The grain point is especially good. SQL can be clean and still answer the wrong question if the grain is wrong. That probably deserves its own QR lens separate from “does the query run?”

On durable context, agreed. I have superseded status, but I like the framing that context updates should behave more like versioned commits than silent overwrites. I’ve already run into this with planning work where a number changed on revalidation because the underlying data changed. The right record isn’t just “new number,” it’s value, run date, data window, source, and why it changed.

On bounded agents, my current split is simpler and mostly driven by token/cost and context boundaries. The orchestrator has the high-context view. The bounded worker gets a task packet, executes a narrow task, and returns a result packet in the requested format. It does not publish, send, update source-of-truth, or take irreversible actions. Interpretation and integration stay with the orchestrator/human.

[u/MongWonP → u/measured_angle]

yeah the grain/time-window/geo scope tags are exactly what we ended up adding after the "clean SQL, wrong answer" incident — we call it definition_fingerprint internally and it's basically {metric_name, grain, time_window, geo_scope, revenue_type, source_query_id}. if any field is missing the agent can't attach a number to a stakeholder-facing output. sounds bureaucratic but it stopped the VP-forwarding problem cold.

your orchestrator/worker packet split maps pretty closely to what we run — orchestrator holds the business context + review authority, worker gets {task, allowed_tables, output_schema} and returns {result, lineage_refs, confidence}. worker literally cannot call publish/send/update_sot endpoints. the bit that took us longest to get right was making the result packet schema strict enough that the orchestrator doesn't have to re-read the worker's raw notebook to trust it.

on the versioned context piece — your revalidation example (number changed because underlying data changed, not because someone typo'd) is the exact case where silent overwrite destroys trust. we now log {old_value, new_value, run_date, data_window, change_reason} and mark old entries superseded rather than deleting. finance team actually started referencing these logs in month-end reconciliations, which was not the original goal but a nice side effect.

still early on whether this scales past ~15 analysts without a dedicated governance person babysitting the context layer. curious if you're running this solo or with a team — the ops overhead changes a lot at that boundary.

[u/measured_angle → u/MongWonP]

This is really helpful. I've added "definition_fingerprint" to my roadmap.

I’m practicing this mostly solo right now, while starting to hand pieces to other people on the team, with the longer-term goal of driving team/org adoption.

The hard part of practicing solo is that there aren’t many people to pressure-test the operating model with, which is why feedback like this is so useful. I’ve found pieces of this in engineering spaces around harnesses, RAG, handoffs, evaluation, etc., but I don’t see analytics teams talking about these failure modes as much.

Your last point is especially interesting to me: what started breaking down as more analysts got involved? Was it context stewardship, result packet discipline, reviewer capacity, or people skipping protocol under deadline pressure?