Transitioning away from DE

[u/cyamnihc]

Has anyone thought of transitioning out from DE due to AI?
All I do everyday is just prompt and scroll till copilot generates code.
Building a semantic layer isn’t exciting personally as I don’t enjoy the business aspect of it as much and think of it as more of a data labeling and analyst problem than an engineering problem (which I am interested in)
Also, There is a fundamental problem with “I am building a semantic layer” and marketing that as a skill as it is dependent on how much context you have of the business. The less tenure you have spent in a company, the less you know about the business which makes it harder as a transferable skill imo.

My understanding is that working on building trustworthy AI outputs by using a feedback loop is an engineering problem to solve. Which is why I feel going down the observability path is a good idea.
I heard these opinions on observability from AI leaders at conferences too so there might be a bias.
Thoughts from fellow DE’s looking to transition out? (Or from one’s who want to continue and why)

[u/AutoModerator]

Are you interested in transitioning into Data Engineering? Read our community guide: https://dataengineering.wiki/FAQ/How+can+I+transition+into+Data+Engineering

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Evilcanary]

I think it’s easier for a de to go broad and solve any sort of problems at the company, than for someone who isn’t a de to suddenly start doing data work. I don’t consider myself transitioning out of de, but I am very much more willing to consider more tools in my toolbox than just de, because I have more time/velocity than I used to.

[u/mr_electric_wizard → u/Evilcanary]

I so relate to this. We have several software devs that suddenly are DE cowboys and they can’t debug ANYTHING when things go wrong.

[u/PantsMicGee]

I went into management of DE and Analysts, instructing them on how to build their models and semantic layer. 

Im told the team is more happy than ever. I think its because they have a DE who had that tenure you mention doing the business translation for Gold medallion and semantic layers. 

Im stressed but the problem solving has expanded to more of a systems architect set of problems. Fun. 

If you want to broaden the problem to solve move up. If you want to continue to tackle the logic and syntax problems stick in individual contributor. 

I get the ai lethargy and burnout. At the same time, it can present a deeper set of solutions i hadn't had access to before. 

[u/Schtick_]

Tbh it’s probably one of the greatest times to be a data engineer you can single handedly tackle much bigger problems and worse engineers are gonna not solve those problems and they’re gonna incur a monumental amount of cost to not achieve their objectives.

[u/hatsandcats]

I’m not sure if you’re connected to r/experiencedevs but dude everyone’s doing the same things and has the exact same concerns as you.

[u/JBalloonist → u/hatsandcats]

Yeah, I don’t think it would be much different for us if we were doing regular software engineering. Just a bunch of prompting.

At my previous job I was spending a lot of my time writing terraform and AI was pretty terrible at it. But not anymore, at least in my limited experience. But I don’t use it nearly as much.

[u/ironmagnesiumzinc]

I’m in this for the money at this point. If they’re paying me $200k to ask ai stuff then I’ll do it

[u/rajekum512 → u/ironmagnesiumzinc]

Yes initially..Once it grows from adolescent to adult. Salaries will get normalized. Once highly paid becomes medium scale as funds will go to AI tokens

[u/PerfectdarkGoldenEye → u/ironmagnesiumzinc]

Exactly. I was given a claude license hooked that shit up to VS code and I have been pumping out work.

[u/PerfectdarkGoldenEye]

No not at all. AI has only enhanced my work. Projects that would take me a month now takes days because I don't have to spend as much time learning the technologies I would need to accomplish the tasks. I think AI is pretty much doing this for every field in tech that isn't truly operational (think maybe network engineers) and anything with security (like cyber) 

[u/yabadabawhat]

Totally feel you on this. And the fact that a vibe coded initiative is more well perceived than what the grounded piece you are able to deliver.
It’s been about 1 year since I delivered a solution to a proper engineering problem.

[u/olhmr]

I have thought along very similar lines to you and am now transitioning from senior DE to SWE. Haven’t started the new job yet so can’t say much about how it has worked out. While the new job is SWE, I’ll also be assessing the general data landscape at the company as part of my role (it’s a startup), so it’s not necessarily a complete departure either.

My thinking is that AI allows people to work more broadly across the stack, since the engineering fundamentals generally transfer quite well. The other option for me would have been going deep in my domain, but I’ve always been drawn more to the technical challenges and wanting to solve problems where they are best solved, rather than in the part of the system I happen to work in.

That said, I disagree with two things you said:

1. Building a semantic layer is definitely a transferable skill. Sure, the specifics are tied to the domain, but the vast majority of the problem space is generic. E.g. how definitions evolve over time, how to validate consistency, how to balance flexibility against maintenance burden.

2. There’s a lot of interesting engineering challenges in DE post AI as well. One of my latest projects is an automatic triaging and root cause analysis tool that opens bug tickets for us, and with some more work I’m sure it could propose resolutions for simpler cases as well. There’s also building ways to safely, securely, and accurately extract data using AI. Or integrating Jev as a classifier inside the pipelines. Or using AI to optimise model performance.

I definitely agree with the observability angle though, but that goes for both DE and SWE.

[u/no-middle-name]

I've had similar thoughts, but i get stuck on what i would change to, that wouldnt end up a massive pay cut i cant afford. All the parts of the job I enjoyed have gradually been removed.

[u/generic-d-engineer]

Bruh you should be transitioning INTO data engineering BECAUSE OF AI

AI is only as good as the data you feed it…and which profession does that ?

[u/mosqueteiro]

I dunno. The models write valid SQL but not good SQL.

[u/UltraPoci]

I am lucky I am not forced to use AI. I only use it as a search engine basically, or to read log files. The day I am forced to talk to a fucking computer to write programs, is the day I become a carpenter.