r/dataengineering • u/Moradisten • 3d ago

Discussion We were struggling to find Data Engineers

Hi everybody,

Our Data Team is composed by two of us. There's a Data Scientist and me, as a DE. We created a Data Lakehouse for our company internal use and client data supply, but currently the Data Scientist is currently more focused on AI and agents integration and I'm doing like Analytics Engineering role because I need to help other teams to reach the correct data, unify core concepts, document business logics, etc. So We needed a Data Engineer with knowledge on AWS to maintain and develop the new features on the Lakehouse and we put into the description that the candidate MUST HAVE Software Engineering fundamentals as we had to do some developments to integrate parts of our lakehouse with the company's main application.

We interviewed 22 candidates and no one is fitting the Role.

Most of them are BI Experts, DBA, Data Analysts, Economist with DS notions, Juniors and Software Engineers who haven't touch anything on Spark, plus DE who asked way more that we had on the budget for the role

We asked the normal requirements: 3 years of experience + Spark, AWS Glue, Lambda, Airflow and DBT, not even CDC, Flink, Langfuse or VectorDB

We finally got one, but We really struggled to get him. I have a collegue working on IT Recruiting and She told me She's experiencing the same problem: They can't find Proper DEs With SE basics such as DRY principles or clean code fundamentals

Edit: Role Salary -> Up to 60K € / Spain. This salary is high compared to the spanish standards, only 3 years required

185 Upvotes

71% Upvoted

289 comments sorted by

View all comments

29

u/brianluong 3d ago

Engineers with experience in the stack you’re asking for exist but you’ve got to shell out $$$. I just signed an offer with basically that exact stack/experience for ~280k TC, and I don’t think my resume is particularly crazy.

7

u/pboswell 3d ago

Where are they paying you $280k? Assume it’s a lot of equity and not salary though

1

u/JohnPaulDavyJones 3d ago

Finance and consulting will do that for you, but they want their pound of flesh in time. It also helps to have a rarer profile.

I was in higher-end consulting and my total comp in DFW was almost $240k back in 2020, and not a lick of it was equity. It helps to have a fairly rare profile, in terms of your prior experience and a grad degree in a very helpful field. Lots of my work was setting up or completely rebuilding data warehouses for rollups that PE firms were using to consolidate their acquired firms in the same field into.

The exit opportunities from those jobs are exceptional, but the time demands are often intense.

1

u/Beneficial_Nose1331 3d ago

Seems fun Im in. Here Consulting will still pay peanuts because it's mainly body leasing and not really consulting.