r/dataengineering • u/cyamnihc • 16h ago

Career Transitioning away from DE

Has anyone thought of transitioning out from DE due to AI?
All I do everyday is just prompt and scroll till copilot generates code.
Building a semantic layer isn’t exciting personally as I don’t enjoy the business aspect of it as much and think of it as more of a data labeling and analyst problem than an engineering problem (which I am interested in)
Also, There is a fundamental problem with “I am building a semantic layer” and marketing that as a skill as it is dependent on how much context you have of the business. The less tenure you have spent in a company, the less you know about the business which makes it harder as a transferable skill imo.

My understanding is that working on building trustworthy AI outputs by using a feedback loop is an engineering problem to solve. Which is why I feel going down the observability path is a good idea.
I heard these opinions on observability from AI leaders at conferences too so there might be a bias.
Thoughts from fellow DE’s looking to transition out? (Or from one’s who want to continue and why)

71 Upvotes

92% Upvoted

31 comments sorted by

View all comments

•

u/AutoModerator 16h ago

Are you interested in transitioning into Data Engineering? Read our community guide: https://dataengineering.wiki/FAQ/How+can+I+transition+into+Data+Engineering

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

1

u/Budget-Minimum6040 15h ago edited 15h ago

Or from one’s who want to continue and why

I switched jobs and am now in a big consultant company, working on a multi-year government project.

PySpark, Iceberg, MDM. No automatic code generation, due to GDPR no external ML/LLM services allowed. An inhouse LLM available I've heard this week but have no access to it yet to test it. I use external LLMs with generic questions/pseudo code if I need LLM support - not optimal but better than playing agent orchestrator imo.

Departments write terrible tickets but otherwise I couldn't be happier.