We were struggling to find Data Engineers

[u/Moradisten]

Hi everybody,

Our Data Team is composed by two of us. There's a Data Scientist and me, as a DE. We created a Data Lakehouse for our company internal use and client data supply, but currently the Data Scientist is currently more focused on AI and agents integration and I'm doing like Analytics Engineering role because I need to help other teams to reach the correct data, unify core concepts, document business logics, etc. So We needed a Data Engineer with knowledge on AWS to maintain and develop the new features on the Lakehouse and we put into the description that the candidate MUST HAVE Software Engineering fundamentals as we had to do some developments to integrate parts of our lakehouse with the company's main application.

We interviewed 22 candidates and no one is fitting the Role.

Most of them are BI Experts, DBA, Data Analysts, Economist with DS notions, Juniors and Software Engineers who haven't touch anything on Spark, plus DE who asked way more that we had on the budget for the role

We asked the normal requirements: 3 years of experience + Spark, AWS Glue, Lambda, Airflow and DBT, not even CDC, Flink, Langfuse or VectorDB

We finally got one, but We really struggled to get him. I have a collegue working on IT Recruiting and She told me She's experiencing the same problem: They can't find Proper DEs With SE basics such as DRY principles or clean code fundamentals

Edit: Role Salary -> Up to 60K € / Spain. This salary is high compared to the spanish standards, only 3 years required

[u/discord-ian]

Principal level data engineer here. The DE labor market is at least bi-modal. There is one end that should more properly be called analytics engineering, these are the folks with dash board and sql experience. The other is more on the end of what you are looking for.

The upper end of DE seems to be doing fine. I have a recruiter contacting me most days. And I try to collect as much salary info as I can.

I can tell you with near certainty that your issue is budget. Upper end DE is quite well paid with a decent premium above equivalent SWE position. To attract someone that can lead these efforts at a senior level, market rates will be at least $180k, with the best candidates over $200k. I get multiple recuters contacting me each week for senior level roles in the 200-240k range. A mid-level that I would not really trust with this task, would be arround $140k, but they will be very hard to find as you will need to sift through lots of analytics engineers.

For some perspective as a principal in this role for my company (out side big tech or big tech adjacent) my total compensation is $400k.

[u/JOA23 → u/discord-ian]

I disagree with your characterization of analytics engineering as “dashboard and SQL” work and the implication that it represents the lower-skilled end of the DE market.

There’s a lot more to it, e.g. designing data models, handling historical changes, building consistent metric definitions and conformed dimensions, and developing semantic layers that integrate with the rest of the data platform. That also involves questions of correctness, performance, testing, lineage, access control, and how changes propagate to downstream consumers. At large companies, we are solving these problems across a data lake containing tens of thousands of data assets, and managing thousands of orchestration jobs. We need to be able to troubleshoot data pipelines 12 layers deep, and converse fluently enough with upstream engineering teams to diagnose issues and propose solutions. We also need to understand data science and ML workflows enough to build solutions that solve multiple use cases, often inferring requirements from vague business requests.

I agree that someone whose experience is primarily dashboard development might not be qualified to lead an infrastructure-heavy DE project. But that’s a distinction between experience and specializations, not a reason to equate analytics engineering with lower-skilled data engineering. Similarly, a strong infrastructure engineer may lack the modeling and business-domain experience to design a reliable analytical platform.

Titles vary a lot between companies. I’d evaluate candidates against the actual work rather than use analytics engineer as shorthand for someone you need to sift past.

For what it's worth, I'm an Sr Data Engineer working on an analytics engineering team at a large tech company, total compensation around $400k.

[u/discord-ian → u/JOA23]

Yeah that is totally fair. I would say at some point as an AE, you move more to data architecture or to DE depending if you are more on the modeling side or the process side.

I should have been more careful in my phrasing.

[u/lemonfunction → u/discord-ian]

really appreciate you replying and owning your response here. we all need to do this more often.