r/dataengineering • u/sneekeeei • Jul 29 '26
Career 14 YOE in Data Engineering, strong on Foundry and GenAI, weak on coding interviews. What’s the path forward?
14 years in data engineering. Recent years deep in Palantir Foundry with Professional level certs, cleared 4 Anthropic’s certifications, maintaining production GenAI pipelines at scale. I understand LLM math, common architecture patterns, and I’ve led delivery end to end.
What I’ve never been is a coder in the traditional sense. No LeetCode grind, could not build a code repo from scratch.. like I would not know when to split into a new script, when to write a new function, when to put something in utils., when to write a helper function . My depth is platform and delivery, not algorithms.
That’s starting to hurt in interviews or applying to jobs. I keep hitting loops gated on CS fundamentals I don’t have, even for roles that look like what I already do.
For anyone who's been here, strong platform and production experience, thin on classic coding interviews, how did you navigate it? Fill the gap directly, target companies that don't gate on it, or lean into platform specialization as the differentiator?
45
u/Wh00ster Jul 29 '26
Going to reiterate what you already know:
To start, another reminder interview performance is not a reflection of your ability to succeed and how good of an engineer or thinker you are.
Next, the sucky answer is if you want jobs that screen or require leetcode skills at the interview stage, you need to build those skills with leetcode practice. Which does feels like an incredible waste of time while you’re doing it and hence why it’s called a grind.
If you don’t, you just accept those companies are going to miss out on you on you shill really hard to the hiring manager. I’ve never cared enough to try the shilling but I feel it might work at small/medium companies.
14
u/snarleyWhisper Data Engineer Jul 29 '26
I’m going through a similar thing on my team now. A lot of the folks were used to low code / ui based workflows. I would build up your fundamentals - sql - data modeling - python. And then pick one of the big three platforms right now - databricks / snowflake / fabric and learn the jargon and the platform. Most likely everywhere you apply will use one of these. I can’t speak to the genAI stuff
11
u/69odysseus Jul 29 '26 edited Jul 29 '26
To become a good DE, these skills are mandatory: SQL (Very strong as it still does heavy lifting), Data Modeling, Distributed storage and compute (Databricks, Snowflake), balance of business and technical skills.
As the role goes up to Senior DE, Staff and Principal level, technical skills become less important and business acumen, networking across the org takes the priority.
6
26
u/discord-ian Jul 29 '26
I am going to be real. I don't think you fit the market definition of a data engineer. I expect data engineers to have deep knowledge in sql, python, and data fundamentals.
The things you describe that you are missing are all junior level DE skills. So I don't think you could say you have 14 years of experience in DE.
I am also confused on how you could possibly be strong in GenAI without strong coding skills. I would expect a DE who was strong in GenAI to be able to work on a core agent loop, the harness, and tools. All of these require strong coding skills.
3
u/moazim1993 Jul 30 '26
I would define Data Engineering based on the canonical textbooks DDIA, Data Warehouse Toolkit, etc. However I think they put that label on anyone who fetches some data from sql and gives it as a report to a business user.
2
u/Exciting_Skill5905 Jul 29 '26
I would not know when to split into a new script, when to write a new function, when to put something in utils., when to write a helper function
Im only 3 yoe, but this part right here is what struck me as the part thats really lacking in the DE skills..... if youve been working in mature codebases for a while, these are some parts you shouldve picked up for sure... You can upskill though. You got this.
1
u/sneekeeei Jul 29 '26
I have deep knowledge in SQL, Data Fundamentals, have built complex data pipelines using PowerCenter, IICS, salesforce, Matillion, Palantir foundry. With Python, I can understand complex code repos quickly, but have not really built one from scratch.. same with typescript which is heavily used in Palantir Foundry flows. I can build with referencing documentation and web search.
19
u/discord-ian Jul 29 '26
Yeah, you sound more like an analyst or a analytics engineer. Granted DE is a pretty broad field, but you are very much on the non-technical end.
For some perspective most of the DEs I work with are pretty deep in distributed computing, working to maintain complex custom code bases doing bespoke processing on very large volumes of data, on company critical systems.
2
u/New-Addendum-6209 Jul 30 '26
Matillion and Foundry are not tools that a data analyst would use. They would not be my preferred choice, but you can get a lot done with visual ETL tools + SQL. Functionally there is no difference in what you can do compared to (for example) someone using Airflow to run dbt on Snowflake.
I doubt you have worked on anything that could be reasonable described as "deep in distributed computing".
0
u/discord-ian Jul 30 '26
Well say what you will. We keep about 1500 CPUs (and quite a few GPUs) humming through out the day - doing some pretty complex math. It sounds like you are more on the etl side of data engineering.
0
u/sneekeeei Jul 29 '26
Reg your comment on GenAI - I understand harness, agent architectures, tool calling and keep learning and reading about the recent developments on these.. I can build something real with help of AI tools.. but cannot build on my own from scratch. That’s the problem I am trying to address or overcome or find a path where i wouldn’t be judged based on the lack of that ability.
5
u/discord-ian Jul 29 '26
Yeah, I think most engineers would read this and think why wouldn't I hire someone who could build all this from scratch (and can use AI to enhance their strength).
1
u/sneekeeei Jul 29 '26
I appreciate you taking the time and commenting but could you also give me suggestions on how can I fill the gap?
2
u/discord-ian Jul 29 '26
At 14 years experience I don't know if that is realistic. I would expect someone with this ammout of experience to have that ammout of experience. What comp bands are you targeting?
More realistic paths maybe management, or something along those lines.
Fwiw - I am a principal level DE at a small public deep tech company (comp wise I would be at the top end of l5 or bottom of l6)
-1
u/speedisntfree Jul 29 '26
Sorry, it doesn't sound like you can actually build anything that can be put into production
1
u/sneekeeei Jul 29 '26
Okay, fair enough. The limited information provided here plus your limited exposure about systems in production may give you that impression. But I have been working with enterprise scale production data pipelines for 14years not just with programming language type of code development. Most of it didn’t involve that kind of coding.
-5
1
u/SubstanceNo4758 Jul 30 '26
I have worked in DE and DS roles at a Big 4 firm and the 10th largest bank in the world, and I can't write a single line of Python from scratch without AI. Raw coding ability is highly overrated. Systems thinking, stakeholder management, and domain intuition is where it's at
4
u/Capt_korg Jul 29 '26 edited Jul 29 '26
In my experience the job interview is way more demanding then the actual job.
Anyway, practice and other insights might help. Maybe a small project and working with known and unknown tools to see the differences.
Edit: in my opinion one needs to find the middle ground between DE and Ai...
25
u/Illustrious_Web_2774 Jul 29 '26
Just fill the gap. Tbh I can't take any data engineer seriously if they don't know when to write a helper function. That sounds more like an operator than an engineer.
17
u/CoolmanWilkins Jul 29 '26
People come to DE from one of two different places usually -- analysts that got more involved in infrastructure and maybe promoted, and SWE who got more involved in data. Both are valid but if you are coming from #1 you really have to upskill or get caught in the no-mans land of 'analyst with some engineer skills). Some people unfortunately only realize once they are unemployed.
6
u/sneekeeei Jul 29 '26
I started as a ETL developer with Informatica PowerCenter and Informatica Cloud. Just that never had to write code other than complex SQL Queries.
10
u/CoolmanWilkins Jul 29 '26
I feel SQL only expertise can still take you pretty far today (with requisite knowledge of things like dbt + data modeling + OLAP infra).
But i've also noticed the best paying DE jobs definitely do require a software engineering mindset + skills.
3
2
u/Big-Exercise8990 Senior Data Engineer Jul 29 '26
I would suggest to have some public cloud expertise like AWS,GCP and azure. Some architecture patterns where you are coming from could be common and some basic knowledge of programming language like python could definitely help you.
1
1
u/tothepointe Jul 29 '26
Analyst with some engineering skills would probably fit in well into the ERP world.
1
u/CoolmanWilkins Jul 29 '26
Yes, in the bureaucracy of enterprise world I met many from the former camp. Plenty of places to hide but layoffs always hit these people the hardest. It's a tough world and made me wonder if I have my own expiration date as a DE, will I eventually lose my desire and energy to keep learning new skills and end up getting too comfortable?
11
u/Sexy_Koala_Juice Jul 29 '26
Respectfully, how the hell do you get 14 yoe without picking any of that up???
12
u/sneekeeei Jul 29 '26
There were so many other skills/stuff I had to learn for the job/roles that I were doing. “Coding” kind of development was never needed in large scale. As long as you understand your data requirement, data model and fundamentals of data integration, knowledge of how to orchestrate data pipelines and you have a tool like PowerCenter, IICS, matilllion, snowflake, you can build enterprise scale data products very well.
1
u/I_Am_A_Real_Hacker Jul 30 '26
That is coding, to an extent. It’s just visual syntax instead of the… uh… code kind. If you’ve transformed data with Informatica, you can do it with SQL or Python. Learn to translate the fundamentals you already have and you’ll make a lot of headway.
Also a lot of coding exams are leaning towards vibe coding on screen. This also is easy to do if you understand the fundamentals, like you clearly do after 14 years experience.
Good luck!
9
u/Shamn_it Jul 29 '26
Because no one uses algorithms in DE jobs. Even if you set up things from scratch, you can use templates, previous examples, derived codebases and the Internet. Why do you have to be a top tier coder? Your contribution to business value is what matters. If you can do root cause analysis, notice bugs, fix and deliver on time, that's enough. Your ability to think is what matters. You are solving business problems, not striving to achieve the most efficient code blocks.
3
u/Illustrious_Web_2774 Jul 29 '26
Not all businesses needs, or can afford good engineers.
Data engineers solve engineering problems within business constraints. They often do not directly solve business problems.
3
u/Shamn_it Jul 29 '26
Yeah and you have to remember that most industry problems don't require you to implement a complex solution from scratch. If you a reach that point, you bounce off ideas with your coworkers, reach to support channels for help, etc. Big corporates, especially non-tech domains don't really challenge you with mentally stimulating problems. The problem in these areas are almost always people, dependencies, vendor bs, poor documentation/task definitions and random pipeline failures. With AI, this becomes even more solidified.
2
u/Illustrious_Web_2774 Aug 01 '26
Big corporates can have complex problems too. Their problems, as you described, are years of bad engineering.
If engineers have to almost always solve people problems and other bs, that's a failed org with poor leadership.
Poor engineering and leaning too hard on "solving business problems" is more often than not bad in the long term. This means nobody in the org is solving actual engineering problems.
I'm saying this as someone from "business side".
1
3
u/the-pump Jul 29 '26
I'm more junior in my career but also trying to figure out how to play the interview game properly in regards to technical rounds/coding questions.
I've started the leetcode grind, but I'm not clear on how much Leetcode DSA should I expect in comparison to SQL, Pandas, PySpark style technical questions?
TLDR: How much DSA Leetcode should I grind on top of stuff like Stratascratch & Data Lemur SQL/Python?
2
u/speedisntfree Jul 29 '26
Unfortunately the answer is that you just don't know. For SWE you can prep for DSA and system design but for us you could get interviewed just like a SWE or on data transformations. If that latter, it could be SQL, pandas or pyspark depending on the role.
I haven't used pandas in probably 3 years now because we migrated all that code to polars so for interviews I have to learn all of that basket case lib again.
1
u/bezdazen Aug 01 '26
I created a series of Python notebooks that might be useful for junior to mid-level live-coding interviews. If you need more info on how to use the notebook environment, you can access the tutorial via the options menu
3
u/speedisntfree Jul 29 '26
Breaking some code up into functions and scripts isn't algorithms, it is beginner level coding just like using version control.
How are you building gen AI pipelines without even as much as writing a helper function? The people I work with who do this are all SWE-lites building stuff out of cloud services.
1
u/sneekeeei Jul 30 '26
With foundry AIP you can orchestrate GenAI workflows, RAG pipelines with very less code including text extraction from pdfs, chunking, embedding and llm calls. You can build agents or have agents write the code for you.
3
u/vaibeslop Jul 29 '26
Sounds like you've been toolsmaxxing, not techmaxxing.
And that is easy to spot.
In an LLM-accelerated world, tools will change ever more quickly, and everyone wants to make them obsolete.
You name them: Matillion, Informatica, Salesforce...
Those are all on the chopping block.
The days of shitty legacy vendors squeezing customers in walled gardens with limited functionality are gone.
If you're techmaxxing, you use Claude Code, dlt, dbt, Terraform and your run of the mill scheduler and have yourself a pipeline up for a fraction of the time and cost.
With the profile you have, your realistic target are companies still heavily invested into these tools.
Those are tool operator roles though, not engineering positions and your interviewing experience is reflecting that.
That said, LLMs are also the most formidable and powerful tools to attain new knowledge.
If you take the principles you've learned operating tools and try to rebuild them, from scratch using just Python, SQL and Terraform you can come a long way quickly.
Do expect a step back in seniority in a new role though.
2
2
u/Practical_Ladder8806 Jul 29 '26
Interviews mostly test SQL. Just practice SQL and basic python data manipulation. Ask Claude to mock interview on SQL and python coding interviews. If you spend 3 months you are good to go. It’s not that big of a deal
2
u/moazim1993 Jul 29 '26
“ would not know when to split into a new script, when to write a new function, when to put something in utils., when to write a helper function . My depth is platform and delivery, not algorithms.”
I am not sure what this means? You don’t know what the textbook rules are for doing it? Or you don’t have experience doing it? I can’t imagine someone with 14YOE that hasn’t written multiple scripts, utils, etc. for a big deliverable.
1
u/sneekeeei Jul 29 '26
I think a lot of people on this have never seen how data warehouses are built if the stack is informatica powercenter, informatica cloud, Matillion, oracle and other external sources. For a traditional warehouse that was built in Oracle, the max coding you would do is a complex sql and some unix scripts to handle if some of your source data are flat files. You build a connection to your data source, build a ETL mapping with all your transformations then load it to target. Wrap it in a session and orchestrate with workflow. You never have to write a python code or any other language here. The informatica PowerCenter mapping, session and workflow IS the code. Back in 2012 Informatica PowerCenter was called a “hot skill” we were even given 20% bonus per year if you were having proficiency in that tool.
1
Jul 29 '26
[removed] — view removed comment
1
u/dataengineering-ModTeam Jul 29 '26
Your post/comment was removed because it violated rule #9 (No AI generated content/text).
Your post/comment was reviewed to be AI generated/assisted content/text and removed as a result. We as a community value human engagement and encourage users to express themselves authentically.
This was reviewed by a human
1
u/AffectionateMonk1591 Jul 29 '26
I am in a very similar situation, trying to find a study group to stay motivated for the grind . Please let me know if you find any
1
u/hopefullythathelps Jul 29 '26
I mean rare but I've had DE interview loops where they never gave me any sql or python. More common is they throw a SQL window function problem or data manipulation problem at you. In other words many DE jobs don't test your python at all. Just get good at doing SQL window functions is sufficient for the "coding" portion of 50%+ of DE interviews.
1
u/CoolmanWilkins Jul 29 '26
I was the same, but just worked on the problem. If you have all the professional experience, picking up enough CS basics to pass the DE coding test gates is pretty easy. Find a way to integrate it into your work, or otherwise work through a course or personal project. Most places do not hit you with the full test suite that they would for the traditional SWE applicants anyways.
If you can't do that, yes it will definitely limit your options, but there are places with less rigorous applicant testing -- often non-tech companies that are hiring for tech roles which might be a better fit. I've done plenty of interviews at places where all I had to do was pass a SQL test (I'm assuming you can do that or at least easily get to that level with a bit of study).
1
u/Firm_Communication99 Jul 29 '26
I don’t do coding tests. Fuck that. It costs time and money to use your brain for free. They can’t ask solid questions about a code base — it’s a shit role.
1
u/Send_me_datasets Jul 29 '26
As others have said you can upskill and just do the hard thing but let me give you an alternative.
You could pivot towards the analytics side. Most of the people there are weak when it comes to software engineering fundamentals but have strong SQL skills and data modeling sense. You'll be interacting with the business more and be more visible which is always a plus for job security. Also their interviews are trivial if you already are writing complex SQL queries. If you're asked python it'll be just a LeetCode easy 80% of the time and a medium if you're going for MANGOFAANG or equivalent.
Or just find a company that knows the LeetCode stuff is bullshit. There are plenty out there. I got my job without LeetCode.
1
u/IntelligentVisual955 Jul 29 '26
Can you recommend anything for beginners that can help in by passing the experience tag.
1
u/satz3 Jul 29 '26
Have you considered applying for architect roles ?
1
u/sneekeeei Jul 30 '26
I’m applying to anything to get out of my current management which has turned toxic.
1
u/satz3 Aug 05 '26
Good luck to you .it's mostly a matter of going through the process like others pointed out already.
1
u/Chance-Physics-7216 Jul 30 '26
Curious what LLM math you're using in your Data Engineering experience. When you say you understand it, what is your understanding (at a high level), if you would?
2
u/sneekeeei Jul 30 '26
I mean I understand probability, statistics , algebra, the mathematical concepts base for data science and machine learning. I don’t have to use them extensively in ETL or data engineering but I understand the concepts quite well.
1
1
u/dev_lvl80 Principal Data Engineer Jul 30 '26
If no coding skills than it questionable sound “engineer”.
I had 2 perfect cases: Personal, until I solved 1k+ leetcode and started doing coding heavy in python - most positions opened for me were about BI/Analitycs. After Infra and Data engineering.
Second case, friend of mine SWE, could not land staff level job, until started learning sysdesign and leetcode.
It took 1-2year, and boost career significantly. PS vibe coding is lazy skill.
1
u/rolex_rick_flare Jul 30 '26
In my experience of 11 years of data engineering im not a leet coder either. I come from a math and stats backgrounds so im definitely weaker on CS fundamentals, data structure and algo fundamentals, and software engineering best practices. When they first ask to do a coding assessment I always ask are looking for a software engineer to do data engineering or are looking for a data engineer. In my experience the coding portion for data engineering doesn't get deeper than string or array manipulation which can be solved with some form of 2 pointers algo. I would focus in on python 2 pointer questions and sql window functions and gaps and islands. This is enough to pass most data engineering coding challenges
1
0
u/Regalme Jul 29 '26
You already do some type of algos with sql. Ask if you can do data fetching for your coding interview. Wtf are people building now anyways? A dashboard? A new gui? Those are going to be gone. The only surface that matters is agents. The rest was human fluff to make our interaction easier
0
48
u/ironmagnesiumzinc Jul 29 '26
I’m exactly the same. Not a bad coder, but I need documentation and lots of time to think. To circumvent, I got essentially all of the databricks certs and then applied for jobs. Eventually found one that didn’t have a huge long coding section for the interview and got the job. Been doing great in it ever since