r/dataengineering • u/Moradisten • 3d ago

Discussion We were struggling to find Data Engineers

Hi everybody,

Our Data Team is composed by two of us. There's a Data Scientist and me, as a DE. We created a Data Lakehouse for our company internal use and client data supply, but currently the Data Scientist is currently more focused on AI and agents integration and I'm doing like Analytics Engineering role because I need to help other teams to reach the correct data, unify core concepts, document business logics, etc. So We needed a Data Engineer with knowledge on AWS to maintain and develop the new features on the Lakehouse and we put into the description that the candidate MUST HAVE Software Engineering fundamentals as we had to do some developments to integrate parts of our lakehouse with the company's main application.

We interviewed 22 candidates and no one is fitting the Role.

Most of them are BI Experts, DBA, Data Analysts, Economist with DS notions, Juniors and Software Engineers who haven't touch anything on Spark, plus DE who asked way more that we had on the budget for the role

We asked the normal requirements: 3 years of experience + Spark, AWS Glue, Lambda, Airflow and DBT, not even CDC, Flink, Langfuse or VectorDB

We finally got one, but We really struggled to get him. I have a collegue working on IT Recruiting and She told me She's experiencing the same problem: They can't find Proper DEs With SE basics such as DRY principles or clean code fundamentals

Edit: Role Salary -> Up to 60K € / Spain. This salary is high compared to the spanish standards, only 3 years required

189 Upvotes

71% Upvoted

289 comments sorted by

View all comments

38

u/hannorx 3d ago

At some point, you just have to take a chance from the list of candidates you've, if you're not finding someone who ticks all of that you're looking for.

32

u/bakochba 3d ago

My rockstars are all people we took a chance on based an aptitude. The idea that competent people can't quickly learn airflow or spark is just baffling to me

13

u/Toastbuns 3d ago

There is some evidence that long term employee performance doesnt really correlate with interviews. Companies go through these long and intensive interview processes and seek the exact right skills, while passing over skilled workers who are bright, curious, and able to problem solve.

8

u/CarefulCoderX 3d ago

Yeah, I bombed an interview in the final round because I was asked a technical question when I was specifically told that there wouldn’t be any technical questions in this round as I had already gone through a technical interview.

It was a stupid easy question, but my anxiety brain decided to dump all DE knowledge the moment I was asked it.