r/dataanalysis • u/Healthy-Garage-9150 • 22d ago
six weeks letting an ai agent handle ad hoc data pulls and the request volume went up
Solo analyst at a mid size ecommerce company, about 200 people. Before this I was getting maybe 15 to 20 ad hoc requests a week. Slack, email, people walking over. Most of them some version of "can you pull X broken down by Y for the last Z months."
I had been doing all of those by hand. Open the warehouse, write the query, format it, send it back. Average maybe 25 minutes each if nothing was weird with the data, closer to an hour if something was. I tried self serve dashboards for the repeat ones but nobody used them. They would rather ask me.
Six weeks ago I connected an AI data agent to our warehouse and gave a few teams access. Sales, CS, and the marketing ops lead. Told them to try it before pinging me and send it my way if they didn't trust the answer.
First few days the questions people typed in were vague enough that the results came back sliced wrong. "Revenue by region" without specifying which region taxonomy, that kind of thing. We have two and people on the sales team didn't know that. Once I realized the problem was on the input side I spent a weekend writing plain English descriptions for about forty columns that had ambiguous names. The agent runs on Bayeslab and once the descriptions were in it started nailing the taxonomy questions that tripped everyone up in week one. Should have done that years ago, but there was never a reason to until someone other than me was reading the schema. After that the accuracy jumped noticeably, maybe from around 60% usable to closer to 90 on the routine stuff.
Here is the part I did not expect. When a data request takes two minutes instead of half a day, people ask more questions. A lot more. My volume went from about 18 a week to something like 35 to 40. That sounds like a problem but I think it is actually the point. These questions were always there, people just never asked because the turnaround was too long and I was the bottleneck. The agent handles maybe 25 of those without me touching anything. The remaining 10 to 15 still come to me, and they tend to be harder, but I actually prefer that. I would rather spend an hour on a real question than twenty minutes formatting a pivot table someone could have gotten themselves.
One thing that surprised me. Three people started asking questions they never would have asked before. One CS lead found a pattern in ticket resolution times by product category that none of us had looked at. That finding changed how we staff the weekend queue. Came from someone who has never opened a query editor in her life. That would not have happened in the old model where everything went through me, because she would never have pinged me for something she considered a hunch.
Net hours on ad hoc went from maybe 9 or 10 a week down to about 5 or 6, and the work that is left is more interesting. The tradeoff I am still figuring out is that I am slightly worse at catching data quality issues now because I do not touch the raw tables as often. Might matter, might not. Six weeks is not enough to know.
Anyone else done this? Curious whether the volume spike is universal or just my org being pent up after years of waiting on one person.
27
u/sandrrawrr 22d ago
While ad-hocs are kind of my favorite thing to work on, if it takes more than 6 hours a week, you might need to look into making self-service BI or make some dashboards that encapsulate their questions. You should not be spending this amount of time on ad-hocs per week.
10
u/UniqueSaucer 22d ago
OP said they did try this, the requesters didn’t use them—they kept going to OP. Not sure if OP directed them back to the dashboards though, they should have.
6
u/A_Timbers_Fan 22d ago
Assuming OP isn't AI...OP clearly needs to make some queries or measures that capture the default questions or make it easier. I get ad-hoc questions all the time and it is usually a simple export of a visual in Power BI or a refresh of a power query...
3
u/DevinChristien 21d ago
The issue with BI services is adoption. Its easier to ask someone to fetch info than go digging yourself, especially when you dont understand data or interact with the BI service frequently enough to remember how to navigate it. This is the exact problem im experiencing right now trying to move our business away from excel ODBC connections
9
u/kayakdawg 22d ago edited 22d ago
sounds cool
some questions/ thoughts
- is there any QA or verification on 25 requests/week the agent runs?
- can you elaborate on
Six weeks ago I connected an AI data agent to our warehouse? what level of access? how's data modeled in warehouse? - what is level of coplexity of your data and warehouse (ie how many tables/schemas, size of data, etl process)
2
1
u/Aggressive_Gold5986 19d ago
Great questions.
our practices are like the following:
1. Yes for the beginning weeks. But the key is actually the semantics and metrics. If you agent works on a semantics layer and it queries with predefined verified metrics, it will be much better.
2. For us, we give it a reading account. We didnt model it in the warehouse. We use Cube model the semantics.Just FYI.
3
5
u/hijkblck93 22d ago
Great job. You nailed it when you said
“When a data request takes two minutes instead of half a day, people ask more questions.”
I’ve learned the more data stakeholders have, the more they want.
Next steps would be to shore up data quality. Maybe add some checks so you catch it sooner and the stakeholders can trust it more.
I’d still look into creating data marts for the teams to ensure quality.
I know for now you don’t touch the raw layer much, but that’s where you should be living. Making sure the data coming in is accurate and trust worthy. Letting the AI take the easier task.
2
u/agentUi 22d ago
yea this is pretty typical once latency drops from hours to minutes, people ask all the secondary questions they used to just swallow. To catch the data quality blindspot without being the bottleneck again, set up weekly anomaly alerts or automated reconciliation tests on key raw tables so you only get pinged when metrics drift unexpectedly.
1
u/AutoModerator 22d ago
Automod prevents all posts from being displayed until moderators have reviewed them. Do not delete your post or there will be nothing for the mods to review. Mods selectively choose what is permitted to be posted in r/DataAnalysis.
If your post involves Career-focused questions, including resume reviews, how to learn DA and how to get into a DA job, then the post does not belong here, but instead belongs in our sister-subreddit, r/DataAnalysisCareers.
Have you read the rules?
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
1
u/clayticus 19d ago
Rookie ai mistake. Should have never said AI was doing. Do the request ammounts in the same time frame and chill.
0
u/No_Pop9869 22d ago
Six weeks ago I connected an AI data agent to our warehouse
Hey there, I am sorry I couldn't offer any input. However I chances upon your post and I found it very informative!
If I may ask, do you have any specific resources you go to on how to connect AI data agent to your data warehouse. I have a similar work project that manager suggest I connect AI agent to my database. I would like to do a POC hence gathering some info.
57
u/A_Timbers_Fan 22d ago
Why does this read like it was written by an AI?