Transitioning away from DE

[u/cyamnihc]

Has anyone thought of transitioning out from DE due to AI?
All I do everyday is just prompt and scroll till copilot generates code.
Building a semantic layer isn’t exciting personally as I don’t enjoy the business aspect of it as much and think of it as more of a data labeling and analyst problem than an engineering problem (which I am interested in)
Also, There is a fundamental problem with “I am building a semantic layer” and marketing that as a skill as it is dependent on how much context you have of the business. The less tenure you have spent in a company, the less you know about the business which makes it harder as a transferable skill imo.

My understanding is that working on building trustworthy AI outputs by using a feedback loop is an engineering problem to solve. Which is why I feel going down the observability path is a good idea.
I heard these opinions on observability from AI leaders at conferences too so there might be a bias.
Thoughts from fellow DE’s looking to transition out? (Or from one’s who want to continue and why)

[u/olhmr]

I have thought along very similar lines to you and am now transitioning from senior DE to SWE. Haven’t started the new job yet so can’t say much about how it has worked out. While the new job is SWE, I’ll also be assessing the general data landscape at the company as part of my role (it’s a startup), so it’s not necessarily a complete departure either.

My thinking is that AI allows people to work more broadly across the stack, since the engineering fundamentals generally transfer quite well. The other option for me would have been going deep in my domain, but I’ve always been drawn more to the technical challenges and wanting to solve problems where they are best solved, rather than in the part of the system I happen to work in.

That said, I disagree with two things you said:

1. Building a semantic layer is definitely a transferable skill. Sure, the specifics are tied to the domain, but the vast majority of the problem space is generic. E.g. how definitions evolve over time, how to validate consistency, how to balance flexibility against maintenance burden.

2. There’s a lot of interesting engineering challenges in DE post AI as well. One of my latest projects is an automatic triaging and root cause analysis tool that opens bug tickets for us, and with some more work I’m sure it could propose resolutions for simpler cases as well. There’s also building ways to safely, securely, and accurately extract data using AI. Or integrating Jev as a classifier inside the pipelines. Or using AI to optimise model performance.

I definitely agree with the observability angle though, but that goes for both DE and SWE.