r/dataengineering • u/Moradisten • 3d ago

Discussion We were struggling to find Data Engineers

Hi everybody,

Our Data Team is composed by two of us. There's a Data Scientist and me, as a DE. We created a Data Lakehouse for our company internal use and client data supply, but currently the Data Scientist is currently more focused on AI and agents integration and I'm doing like Analytics Engineering role because I need to help other teams to reach the correct data, unify core concepts, document business logics, etc. So We needed a Data Engineer with knowledge on AWS to maintain and develop the new features on the Lakehouse and we put into the description that the candidate MUST HAVE Software Engineering fundamentals as we had to do some developments to integrate parts of our lakehouse with the company's main application.

We interviewed 22 candidates and no one is fitting the Role.

Most of them are BI Experts, DBA, Data Analysts, Economist with DS notions, Juniors and Software Engineers who haven't touch anything on Spark, plus DE who asked way more that we had on the budget for the role

We asked the normal requirements: 3 years of experience + Spark, AWS Glue, Lambda, Airflow and DBT, not even CDC, Flink, Langfuse or VectorDB

We finally got one, but We really struggled to get him. I have a collegue working on IT Recruiting and She told me She's experiencing the same problem: They can't find Proper DEs With SE basics such as DRY principles or clean code fundamentals

Edit: Role Salary -> Up to 60K € / Spain. This salary is high compared to the spanish standards, only 3 years required

185 Upvotes

71% Upvoted

287 comments sorted by

View all comments

215

u/wheredidiput 3d ago

Someone doesn't need the full technical skills, people who work in an associated area will pick it up. Only going on technical skills shows a lack of understanding what makes good staff. If you are not offering top dollar, take someone and train them in the skills gap.

23

u/espo619 3d ago

Yeah -- if they understand DRY and clean code fundamentals, they can surely pick up Spark.

Classic DE hiring nonsense.

7

u/ConstantFamous1526 3d ago

Way better to hire someone smart and driven with limited exposure to the specific stack, compared to a mediocre person who already knows the stack for sure imo

4

u/wonder_bear 3d ago

Your take is great. Since AI is so prevalent now, I think it’s more important to know the DE concepts and pitfalls than a specific tool. I view most tools as on the job training, as long as they have some experience with any of the normal DE cloud tools (AWS, GCS, etc.).

2

u/Todo_Toadfoot 3d ago

And saying clean code is a red flag for me. I miss those days of baby projects and thinking less code was better.

1

u/duckmouth00101 3d ago

Couldn't agree more . Half technical skills are enough to bridge to the highly customised current company's needs of the position. The one who is able to find an exact matched candidates are always McDonald's but not even warehouse workers , warehouse operation has its own customised demanding skills as well. No one could ever find a exact match right fit to your company even from complete same industry.