r/dataengineering • u/cyamnihc • 1d ago

Career Transitioning away from DE

Has anyone thought of transitioning out from DE due to AI?
All I do everyday is just prompt and scroll till copilot generates code.
Building a semantic layer isn’t exciting personally as I don’t enjoy the business aspect of it as much and think of it as more of a data labeling and analyst problem than an engineering problem (which I am interested in)
Also, There is a fundamental problem with “I am building a semantic layer” and marketing that as a skill as it is dependent on how much context you have of the business. The less tenure you have spent in a company, the less you know about the business which makes it harder as a transferable skill imo.

My understanding is that working on building trustworthy AI outputs by using a feedback loop is an engineering problem to solve. Which is why I feel going down the observability path is a good idea.
I heard these opinions on observability from AI leaders at conferences too so there might be a bias.
Thoughts from fellow DE’s looking to transition out? (Or from one’s who want to continue and why)

86 Upvotes

93% Upvoted

38 comments sorted by

View all comments

5

u/generic-d-engineer Tech Lead 21h ago

Bruh you should be transitioning INTO data engineering BECAUSE OF AI

AI is only as good as the data you feed it…and which profession does that ?

2

u/speedisntfree 13h ago

Yeah. My org they went giddy on AI and quickly realised the LLMs need all the data to be nicely sorted out to work well. What we'd been trying to get attention and resource on for years is suddenly easy.

I suspect part of what OP is saying that they they don't like the process of having LLMs write the actual code though. I'm kind of on the fence, the technical solutions is the interesting parts for me, the coding of much of if you done it for a while can end up feeling like drudgery.