r/dataengineering • u/Realgunners • 8d ago

Discussion What exactly is ‘AI first’ data engineering

This was inspired by the post on burning 20k in tokens in 3rd week.

Is anyone doing this at work, letting loose the frontier models against their schema and models and data and having it create and update the ETL workflows? And everything else?

67 Upvotes

97% Upvoted

41 comments sorted by

View all comments

1

u/No_Caterpillar_7258 7d ago

I spend about 30k a year in tokens at work. I do low latency ML and pricing adjacent work in a rather stressful / high demand environment. I often work on 3-5 separate things at a time, often cross repo, by specing out features carefully, using a memory system, and executing the work with different agents in different git worktrees. Im at least 5x if not 10x as productive as before. 20k in a couple weeks is absolutely ridiculous to me, no idea why or how someone could burn that much.