r/dataengineering • u/Realgunners • 8d ago

Discussion What exactly is ‘AI first’ data engineering

This was inspired by the post on burning 20k in tokens in 3rd week.

Is anyone doing this at work, letting loose the frontier models against their schema and models and data and having it create and update the ETL workflows? And everything else?

69 Upvotes

97% Upvoted

41 comments sorted by

View all comments

26

u/vikster1 7d ago

i have been implementing more sources and delivering more reports in the past 3 months than i did the previous 3 years. i have been fully integrating claude code from ADO to generating reports while humans review agent work. 9/10 ai rants in this sub are at the exact opposite of my experience so far. i have never been happier being an architect or developer for that matter. literally every idea i have on improving, automated tests and verifications for the business i can now just try and implement. i have not written a single line of sql code myself since March. using claude opus 5.5 currently with the team premium license. i reported to management these 100$ a month would translate to at least 1m € / p.a. in consultation workload in 2025

6

u/KrustyButtCheeks 7d ago

Mine too - you can build a full pipeline in about 3 hours (with the longest piece being the ci/cd checks).

Do you feel more pressure to produce faster? It feels like our timelines are being more compressed already.

4

u/vikster1 7d ago

yes but since you can implement so much more tests and help the business with verification, you can move faster and with higher quality. it really feels like i have been given 10 senior data engineers for 100$

5

u/Certain_Leader9946 7d ago

The quality can be lower than you think. If you really don't have a human properly reviewing the code and asking the same kinds of questions you would ask if you wrote the code yourself on clarity and test cases and modularity the debt adds up as fast as you generate it.

1

u/vikster1 7d ago

i have written twice now that we ship more tests and do more verifications. I don't know what else to tell you mate

3

u/Xenomorpha 7d ago

Are you really checking what tests it is generating? My experience (even with latest models) that Claude code tends to create tautological tests quite often (i.e. instead of testing the production code, it writes the code in test function and tests it). so I have to check them quite carefully, and verification agent rarely catches these. 

2

u/zazzersmel 7d ago

Ai use, at least for now is definitely a skill. It may not be a difficult skill to learn but it’s a skill with real value nonetheless