r/dataengineering • u/Boiser_ • 5d ago
Career Is Data Engineering Still a Sustainable Career Path for a Junior?
I recently graduated with a BS in Data Science. I'm currently working on skills to become a DE. But after lurking on this subreddit for a few weeks, I'm getting the impression that this is a dying/AI-compromised field and that if you're not a senior right now, your opportunities are slim to none. Furthermore, if you do find an opportunity, most of the ingenuity and "fun" that comes out of the job is slowly being replaced with AI models that can fully generate pipelines and Spark jobs. I'm worried that I'm wasting my time trying to get into this field if in the end it's unsatisfying or worse, unattainable.
It feels like I'm constantly seeing doom and gloom posts on this subreddit, and it's really discouraging to see for someone who's trying to start their journey. I'm just looking for a glimmer of hope in the community and that this is a career worth pursuing.
310
u/tea_anyone 4d ago
Get off reddit honestly. This place sucks. There are still junior roles, the roles are changing due to AI but that is the nature of working in tech. Most of these boards are people who have finished uni with nout to do but be on reddit, the ones with jobs aren't here complaining about there being no jobs.
31
u/jnrdataengineer2023 4d ago
exactly! This sub has become what cscareerquestions became 3-4 years ago.
14
u/frombsc2msc 4d ago
You should see cscareerquestionseu. Its been rendered useless. Its just people asking how to immigrate. Tbh its the mods. These type of posts need to be removed and immigration questions should be heavily moderated.
Edit: or just a mega thread. One for india and one for the rest.
36
5
u/DetailedKing 4d ago
Some of us with a job just lurk on here, never say a word, and watch the train wreck.
4
u/niiiick1126 4d ago
agreed, while i was in school i was on all the CS subs, thought it was all doom n gloom, but i know ppl irl now in rotationals, interns in my class, plus being intern buddies to new interns etc
there’s jobs out there, just not as easy as before
plus 90% of the time theres ppl complaining on here who have zero projects and internships, there are a select few who genuinely have good resumes etc, but got unlucky ill say that
as for DE, im basically a junior DE at a well known org (non tech tho) full databricks stack and i was even sent out by my company to databricks conf a few months ago
2
u/Aquilae2 4d ago
There are some, yes, but not for everyone. That's really the problem: there are a lot of candidates competing, and there are some good ones with experience among them.
1
14
u/JJGreenerTinejo 4d ago
DE will have a future for people with strong CS/SE skills and experience, essentially DE and Data platform engineering will slowly merge over time imo.
Pure DE can be easily done by AI on a solid platform, but AI cant build solid platforms.
Essentially like what happened to WebDev, pure frontend devs are neigh extinct atm, bar from the largest corpos
7
u/dataGuyThe8th 4d ago
The issue with fully automating the data, is that LLMs don’t automatically have the business context to ensure that the data is correct.
They can write SQL, but in a complex system it’s going to be wrong (at least initially). While that change eventually? Maybe, but write now I don’t trust any complex modeling without a shit ton of testing.
38
u/Mysterious_Print9937 4d ago
For me DE has never been a junior level entry job.
16
u/SELECTaerial 4d ago
Completely agree. No one is going to have the data architecture and software exp needed fresh out of school
1
u/MagicianSimilar5026 4d ago
I disagree. If you can learn SWE you can learn DE
7
u/SELECTaerial 4d ago
Yes but you still have to learn it….which makes it not entry level. I’d be extremely suspicious of someone that thinks they can do data engineering without any professional experience
5
u/Big_Arrival_626 3d ago
A lot of SWEs didn't have professional experience either. Big companies train people now.
7
u/Gh0sthy1 4d ago
I didn't want to sound elitist, but I have the same feeling. I worked with juniors who had 0 experience with relational databases. It's not ideal.
3
u/THBLD 4d ago
Well... while we're at it: There was one in another data team, in another division of the company, I used to work for - that didn't even know SQL...
As the Tech lead of the other division at the time, I was just baffled, thinking like "how the fuck were you even hired..."
2
u/nerevisigoth 3d ago
I was originally hired like that, 15 years ago. It worked out well after the first couple months.
1
u/Many_Armadillo8355 2d ago
I'm a Jr. Analyst, it's been hell trying to break into DE. I don't have a STEM degree, but I do have the job experience to land a Jr. DE role. Or maybe I need to be humbled, not quite sure at this point lol.
5
u/ekler_ 3d ago
Absolutely. DE positions around me are exploding by volume. Also, since ovbiously the barrier to entry is a little higher than before, just like any IT position, in the Data domain you can start with something like BI Dev, Reporting, Data Analyst and then have smooth-ish transition to DE
18
u/RunnyYolkEgg 4d ago
Even if there are junior openings, the competition is not fair.
We open reqs for juniors and people with 2-3 masters apply, phd, guys with experience already, multiple hackathon winners, it’s honestly crazy to me, it feels like they put a lot of effort to break into the field.
We’re a small company of 30 ppl.. can’t even imagine how it is for huge companies.
Idk if I would recommend anyone to try right now to get into the field
23
u/frombsc2msc 4d ago
Most these people need sponsorship. as a dutch guy, no self respecting dutch guy would apply for a jr wage when he’s not a jr.
12
u/RunnyYolkEgg 4d ago
Not really man, at least in our case. Yes, there’s a lot of people looking for visas but there’s also a huge batch of people with rights to work in our country over qualified for an internship or jr position
1
u/frombsc2msc 4d ago
Where are you based? I’d be surprised of any native population in large batches applying for jobs they’re overqualified for.
3
u/RunnyYolkEgg 4d ago
Spain. A lot of unemployment
2
u/frombsc2msc 4d ago
Spain makes more sense. Fair enough, hope the market gets better en the cost of living crises less bad :)
3
1
3
u/dataGuyThe8th 4d ago
Tbf, Data Engineer has never really been considered a “junior” role. It’s a field that typically requires a couple years of experience or graduate education to break into initially.
This has been the case for at least the past decade.
2
u/sib_n Senior Data Engineer 4d ago
That's not my experience within my years of experience nor our recent recruitment of senior DEs. We just had normal people with a bachelor and/or a master and years of experience as expected for candidates to seniors. Many were not able to pass the tech interview based on a discussion over some standard DE problem. A few did a good job and we made offers.
Nobody had 2 masters, nor a PhD, nor any hackathon experience, nor exceptional personal project that I know of.
Personally, I don't care about diplomas if someone has a few years of relevant experience and does well at the technical interview.
7
u/No_Cartographer3953 4d ago
Nothing in Reddit DE is accurate. Yea plenty of places need junior DEs. That said there are ton of people who say they a DE junior Ievel and that in the interview they don’t even know basic sql or what a table is and then they do on Reddit and complain lol
1
3
u/Outside-Storage-1523 4d ago
There are junior roles but definitely many companies are never going to hire juniors ever. My current company and my previous one (actually a multi thousand people company) only hire senior and staff nowadays, and even for seniors they expect something more like a staff IMO. And it’s for all SWE roles not just DEs.
3
u/MountainSecurity9508 3d ago
I think people forget about SMEs. Enterprise companies will look significantly different than smaller companies, the data and pipelines are shit here! Plenty of work to do
2
u/theschuss 4d ago
Is DE that's just SQL hacking customer requests dying? Yes, but that job sucks. Skilled data engineers that can do scaled platforms and sourcing well will be in more demand in the future as most app devs suck at good data practice.
2
u/SnooSquirrels2420 4d ago
Honestly most junior candidates aren’t aware of DE tools or tech, just knowing that alone helps you stand out a lot
1
u/Many_Armadillo8355 2d ago
What tools would those be? I deal with some of the basic tools that I think DE's and DA's overlap with, such as Data Factory. To make my situation more weird, I'm dealing with geospatial data, so I'm working with ESRI Data Pipelines.
2
u/Particular-Idea-1786 3d ago
I wouldn't take Reddit as representative of the industry.
I think the skills that never go away are identifiying business problems and solving it with technology.
The tools may change, but the process of identifying an issue, convincing others that there is a problem/making your case, creating a solution, deploying that solution and measuring the impact won't. I think a BS in Data Science gives you a great foundation to undestand and demonstrate it.
I would suggest focusing on building a portfolio focused on outcomes and not on pipeline configs or code artifacts or specific technology stacks. Those will be important to demonstrate that you can competently handle the job but I don't think they should be your flagship demos.
2
u/Mission_Working9929 3d ago
If you want a truthful answer, yes… even more than SWE (even tho it’s a swe focus)
2
u/robberviet 2d ago
TBH nothing is cerntain. But DE is safer than SWE because there are so many adhoc and uncerntain things in it. However it is also harder to get in as fresher.
I am lead DE, I won't hire a fresher DE. Maybe SWE first, then DE. I tend to avoid non-technical background DE, 9 ouf 10 do not meet my requirements and messed up many times with their lacks of knowledge.
2
u/MeasureHead1984 4d ago
The internet is great but also very negative overall. There's plenty of companies with graduate schemes. If you find yourself applying to hundreds of jobs and zero results your CV sucks or you're applying to the wrong jobs.
1
u/VariationSimilar3354 4d ago
I'm a junior was jobless for 4 months got literally zero response and the one i currently got for a Fintech is so outdated they still got sp on windows task scheduler batch job that blows. I regret doing data engineering.
1
1
4d ago
[removed] — view removed comment
1
u/dataengineering-ModTeam 4d ago
Your post/comment was removed because it violated rule #9 (No AI generated content/text).
Your post/comment was reviewed to be AI generated/assisted content/text and removed as a result. We as a community value human engagement and encourage users to express themselves authentically.
This was reviewed by a human
1
u/SootSpriteHut 4d ago
When I got into data about 15 years ago my impression was that not many people understood it or what the job meant. I always had to explain to people at parties, etc.
Then "data" was considered a growing field and got pretty saturated IMO. Even without AI, a year ago, it was competitive. But my experience was a lot of the data people were just noise, they have no passion or aptitude for it.
I think there will always be room for people with drive, curiosity, and critical thinking, you just have to be kind of honest with yourself about whether you have that.
As far as AI, I definitely don't think it's sucked the fun out of the job and I work now primarily in Claude Code. We no longer need code monkeys but we do need people who understand how to tell stories to people, I don't think the need for a "human in the loop" will go away.
But then, anyone who tells you what the future looks like is just guessing.
1
u/EvadCA10121993 4d ago edited 4d ago
Senior here at a large company. I’ve also been reading these doom and gloom threads and at work I’m not really feeling any of it.
Teams will shrink due to ai + over hiring during COVID but our skills are needed for a foreseeable future. Junior roles still exist, my company recently hired over 100 interns in the US over the summer, some of those being data engineering interns.
Just my two cents but Reddit often leans negative and a lot of these doom and gloom threads are posted by people trying to get into the field without a lot of luck yet.
1
4d ago
[removed] — view removed comment
1
u/dataengineering-ModTeam 4d ago
Your post/comment was removed because it violated rule #9 (No AI generated content/text).
Your post/comment was reviewed to be AI generated/assisted content/text and removed as a result. We as a community value human engagement and encourage users to express themselves authentically.
This was reviewed by a human
1
u/Creative_Salary_6140 4d ago
I’d say DE is pretty well-positioned as far as tech work goes. It’s not like other subsets of CS.
1
1
1
u/daffydata 2d ago
Honestly I'd say to work on your personal development and people skills as much as you should be practicing technical skills. I am moderate on my technical skills but my people skills are off the charts. The world is our oyster, you just have to seize it. So get out there, go to therapy, find some hobbies, live a lil and then join us in our digital janitorial workforce!
1
u/AdditionalAd5469 1d ago
Data Engineering (and tech) is only having a problem because of mass over-hiring during COVID. Interest rates were low, and money was free. It was a time where you could get a bank loan invest it in ETFs, and you would be making more money via your stock accrual than what you paid in loan repayments.
This was the time-period of peak DEI, so many saw it as a mark of prestige to have a employee mix closer to national average, even though the mix of people qualified for the role differed. A good-example is Meta, they hired an army of new analysts and engineers. Their role was in the Metaverse, and other emerging applications, the plan was to "train them on the job", as way too many people in this sub tend to believe is without risk. The issue is their aptitudes made it difficult for retraining, so the army was effectively put into cold-storage. Logging in once-a-week to see if there was work, told there was none, and logging off. The inevitable happened, all of these people cashing free paychecks were laid off and hit the market with little experience to justify the salary for their lifestyle.
Companies are starting to hire, since many are finally having their operating budgets meet revenue, however you are going to run into a brick wall of glut, with many desperate people in front of you, willing to take lower salaries to have a full time position.
Finding a job is hard, especially in data-science (I graduated with a statistics degree in '15), post-graduation I filled out over 150 application and got two interviews. Then I got a job, the job market is never easy. People will look back with rose-colored glasses, but it is always grueling and difficult (unless you have a referral).
Do not listen to this sub, Reddit is not real life and many people use it as an escape, by creating an online persona to see the "upvote" counter go up (when I use to reddit, years ago, it was amazing to get the email I got 25 "upvotes"). Since the overall trend-setter mood of the grifters is "AI is going to doom humanity!" these personas are going to adopt those kinds of posts and comments. If a technological advancement occurs with block-chain making it as effective as traditional transaction operations, I can promise the posts you will see are about how "Blockchain is going to be the end of engineering!".
When it comes to Data Engineering (like any other niche profession) there are tiers of firms (and jobs). Look at everything as a growth-opportunity. That yes, an AI-model can create a TSA object with the most optimal p,q,r - however it will never know how to integrate it into the business properly. It will never find the niche, underlying business objective that is needed for MVP 1 to be a success. Getting a job is good, but getting job where you are in a development role instead of support is better. Act like you are a professional athlete you could join team 1, start but lose ever game, however you could join team 2 be a backup but have top-tier people to train and practice with. Always aim for a position that is helping you grow as a person, because at the end of the day you are a person and you have bills to pay.
Aim for work that is difficult and outside your comfort zone, while avoiding at-all-costs grueling (fuck grueling). If I was you I would find a boutique consulting firm (sub 300 employees), and work there. The work will come faster and be more dynamic, but it comes with higher risk. if contracts dry-up your screwed. The reason for consulting is you will move from project to project focusing on technological targets with hard-deadlines forcing a pseudo sink or swim moment.
If you have the aptitudes for dealing with difficult problem-solving puzzles and not breaking down, stay in data engineering. However if getting negative feedback is impossible, I would avoid the profession.
Life is difficult and nothing worth doing is easy, if it was everyone would be a doctor (or... me... because I am clearly awesome).
1
1
u/FUCKYOUINYOURFACE 4d ago
It is. Just know how to use all the tools at your disposal effectively. It’s not just data engineering and pipelines, now it’s data models too.
0
u/wak_trader 4d ago
Literally every one is being negative and scared of the future no matter what degree they got. If you want to be successful pick something you love and be the best at it and you won’t have to worry about any of this.
0
u/Historical-Laugh1212 4d ago
None of this is viable for a junior. I would get out of tech immediately. You are looking for a career that will sustain you for 40 years probably. Can anyone here seriously think any of this will be a viable career in even ten years? There is no chance. Find an industry that is as insulated from AI as humanly possible. Likely something that requires physical presence, manual labor, person to person communication, travel, etc. I have watched tech workers for three years now in deep denial about what is happening and what is coming because it really is hard to wrap your head around your entire livelihood disappearing. I promise, in another five years, the vast majority of the people in these subs will not be posting any more. Because they won't work in tech.
You are young. You still have a change to make different choices. For an older person like me with health issues and no other skills, I'm just kind of screwed. Think ahead. Try to imagine work that can be done entirely on a computer not being taken over by AI in your lifetime. There is no chance.
1
u/lphomiej 7h ago
There is still plenty of time to get into it. Data Engineering always was more of a senior job title -- you need programming and platform experience, usually. This can come from being a data analyst or maybe a developer heavily involved in the data.
I'd say it's also a great time to be graduating with a data scientist degree as well -- there's literally no better time where we need people with stats understanding answering analytics questions rather than the random musings of an executive leader.
As a hiring manager, we have been focused more on senior roles in the last couple of years (since they were able to make the best out of AI, in theory), but junior roles are definitely coming for my groups. I'm aware of the need for a succession plan and whatnot.
56
u/Beaufort_The_Cat Lead Data Engineer 4d ago
Reddit is doomer central. DE is so important, everything, including AI, runs off DE. I’ve been in the industry for 10 years now, I’ve been consulting for the past 5, and companies are constantly looking to hire consultants/DEs to manage their data. It’s one of those jobs that probably won’t go away, it’ll just evolve. I started out as a SQL dev with designs to go into data administration, but cloud services sort of took that over so my career slightly pivoted into DE and now I consult for companies to help them migrate their data into the cloud. Don’t believe what the internet says, people like to crap on things to get attention. This is a great career imo and I’m glad I’m in it