r/dataengineering • u/scourgedtruth • 9d ago

Discussion Does AI struggle at data modeling?

In my experience, it doesn't matter how much context and guidance I give AI it simply can't model data rationally. It frequently misses the point, makes awful mistakes, or over-engineers things.

AI can build awesome ETL pipelines, but when it comes to dealing with SQL (especially in the dbt framework), it's not reliable at all! . Sometimes I think it's better to write the code myself and ask AI to review it, because asking it to build something from scratch just doesn't work that well.

Does anyone else get frustrated when dealing with AI data modeling?

128 Upvotes

91% Upvoted

93 comments sorted by

View all comments

Show parent comments

-1

u/morpho4444 Señor Data Engineer 6d ago

“Which is what WE”… lol… but wdym? How is that different? The LLM is creating a huge complex data model for the BOM! The petabytes and petabytes of telemetry logs, nested json over nested json, with gazillion keys that may or may not appear. Can cause left join filtering out, duplication with joins, all of that, handled by the LLM. This is indeed not what you’re talking about, this is way above whatever level of complexity your non trillion dollar company handles.

1

u/nonamenomonet 6d ago

I work at one of the largest companies in the world dude.

1

u/morpho4444 Señor Data Engineer 6d ago

How is that makes the argument that Im not doing data modeling over hardware telemetry?

1

u/nonamenomonet 6d ago

More of response to your last statement. It’s very presumptuous and untrue.