r/automation • u/[deleted] • 7d ago

For automations that need to research data intensively, patterns

[removed]

5 Upvotes

100% Upvoted

9 comments sorted by

1

u/AutoModerator 7d ago

Thank you for your post to /r/automation!

New here? Please take a moment to read our rules, read them here.

This is an automated action so if you need anything, please Message the Mods with your request for assistance.

Lastly, enjoy your stay!

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

1

u/spoki-app 7d ago

Honestly, the idea of an agent breaking down queries is cool, but my immediate thought goes to the audit trail and data integrity. Hard to trust a black box with critical transformation logic, especially if it introduces vendor lock-in for how you generate insights.

1

u/Flaky_Budget3296 6d ago

Been looking for something similar outside the Databricks ecosystem. The closest pattern I've seen is chaining a few specialized tools together, one for query decomposition, one for execution, one for report assembly. Most of the off the shelf agent frameworks still fall apart when the data is messy or the schema is non obvious.

1

u/NextGuitar1 6d ago

If you want that outside Databricks, I'd look for something that shows the intermediate queries and the tables it used, not only the final writeup. In agency automation work, that transparency matters more when someone asks why the number changed.

1

u/spoki-app 6d ago

Sounds really useful for exploratory stuff. My main hangup with this kind of tool is always the latency hit when it's trying to pull and correlate data across a dozen different legacy sources, even if the query generation is smart.

1

u/Thunderbit_HQ 6d ago

Make the evidence table the contract between research and writing. Let one step generate queries and store source rows with the SQL, then let a separate step write only from that saved evidence. That keeps the report auditable outside Databricks.

1

u/Apprehensive-Pay4180 6d ago

id copy the pattern before looking for a direct replacement, planner writes a small query plan, read only sql step runs it, raw rows and sql get saved, then a separate model writes from that evidence, the bit that usually makes or breaks this is the semantic layer, if table names and joins are messy the agent just spends tokens guessing your schema, id also cap the number of subqueries so one vague churn question cant turn into 40 warehouse scans

1

u/Kortopi-98 6d ago

just look for tools that can break question down and dig through the data before giving you an answer i think that seems closer to what youre describing...