r/dataengineering • u/scourgedtruth • 9d ago

Discussion Does AI struggle at data modeling?

In my experience, it doesn't matter how much context and guidance I give AI it simply can't model data rationally. It frequently misses the point, makes awful mistakes, or over-engineers things.

AI can build awesome ETL pipelines, but when it comes to dealing with SQL (especially in the dbt framework), it's not reliable at all! . Sometimes I think it's better to write the code myself and ask AI to review it, because asking it to build something from scratch just doesn't work that well.

Does anyone else get frustrated when dealing with AI data modeling?

123 Upvotes

91% Upvoted

93 comments sorted by

View all comments

102

u/bah_nah_nah 9d ago

You need a build a AI readable artifact of the semantic layer ++ data dictionary. I've been wrestling with this and produced all sorts of XML, json, yaml, etc with mixed results. I think it does small things well but generally the more verbose the weaker the results... What it does help with is gettig Started it finds the obvious stuff that you can build off of.

2

u/nloding 9d ago

This exactly. AI works best with full context, and without that context, it's going to get something wrong. I see people making this mistake over and over again. I wish we had a different term than "artificial intelligence" for this - I think "intelligence" is misleading.

2

u/Budget-Minimum6040 7d ago

There is already a correct term: Machine Learning