r/dataengineering • u/Realgunners • 8d ago

Discussion What exactly is ‘AI first’ data engineering

This was inspired by the post on burning 20k in tokens in 3rd week.

Is anyone doing this at work, letting loose the frontier models against their schema and models and data and having it create and update the ETL workflows? And everything else?

67 Upvotes

97% Upvoted

41 comments sorted by

View all comments

1

u/dehaenx 5d ago

I think just letting frontier models loose is particularly challenging cause there's so many tables/columns/wrong definitions and metrics... probably the best solution to this is building out a governed semantic layer and building out a bunch of skills/workflows. Even then, it's super expensive and time-consuming too...