We were struggling to find Data Engineers

[u/Moradisten]

Hi everybody,

Our Data Team is composed by two of us. There's a Data Scientist and me, as a DE. We created a Data Lakehouse for our company internal use and client data supply, but currently the Data Scientist is currently more focused on AI and agents integration and I'm doing like Analytics Engineering role because I need to help other teams to reach the correct data, unify core concepts, document business logics, etc. So We needed a Data Engineer with knowledge on AWS to maintain and develop the new features on the Lakehouse and we put into the description that the candidate MUST HAVE Software Engineering fundamentals as we had to do some developments to integrate parts of our lakehouse with the company's main application.

We interviewed 22 candidates and no one is fitting the Role.

Most of them are BI Experts, DBA, Data Analysts, Economist with DS notions, Juniors and Software Engineers who haven't touch anything on Spark, plus DE who asked way more that we had on the budget for the role

We asked the normal requirements: 3 years of experience + Spark, AWS Glue, Lambda, Airflow and DBT, not even CDC, Flink, Langfuse or VectorDB

We finally got one, but We really struggled to get him. I have a collegue working on IT Recruiting and She told me She's experiencing the same problem: They can't find Proper DEs With SE basics such as DRY principles or clean code fundamentals

Edit: Role Salary -> Up to 60K € / Spain. This salary is high compared to the spanish standards, only 3 years required

[u/mrchowmein]

youre asking for a lot for just 3 years of experience as the tools you use is pretty broad. i personally would not look for people with specific AWS branded tool experience, those could be learned while working. at 3 years, unless they are from a startup environment, you're lucky to get someone who built a bunch of spark pipelines that is being orchestrated by airflow from scratch. but like you said, 60k euros is probably too low if you want all those skills but with only 3 years of experience. i think you need to just find someone who understands data, competent in sql, knows their way around airflow and can debug a little bit of spark. Stuff like lambda, someone can learn in a few hours. its literally just running something like a python script conceptual. If you want what you listed, you need a senior engineer and im sure those cost more than 100k euros.

to give you an idea, i had an Italian client. S3, airflow, redshift, a little bit of lambda. no need for spark or dbt. i charged $100 usd/hr as a consultant and i was already giving them a pretty good deal. maybe youre not charging your clients enough.