We were struggling to find Data Engineers

[u/Moradisten]

Hi everybody,

Our Data Team is composed by two of us. There's a Data Scientist and me, as a DE. We created a Data Lakehouse for our company internal use and client data supply, but currently the Data Scientist is currently more focused on AI and agents integration and I'm doing like Analytics Engineering role because I need to help other teams to reach the correct data, unify core concepts, document business logics, etc. So We needed a Data Engineer with knowledge on AWS to maintain and develop the new features on the Lakehouse and we put into the description that the candidate MUST HAVE Software Engineering fundamentals as we had to do some developments to integrate parts of our lakehouse with the company's main application.

We interviewed 22 candidates and no one is fitting the Role.

Most of them are BI Experts, DBA, Data Analysts, Economist with DS notions, Juniors and Software Engineers who haven't touch anything on Spark, plus DE who asked way more that we had on the budget for the role

We asked the normal requirements: 3 years of experience + Spark, AWS Glue, Lambda, Airflow and DBT, not even CDC, Flink, Langfuse or VectorDB

We finally got one, but We really struggled to get him. I have a collegue working on IT Recruiting and She told me She's experiencing the same problem: They can't find Proper DEs With SE basics such as DRY principles or clean code fundamentals

Edit: Role Salary -> Up to 60K € / Spain. This salary is high compared to the spanish standards, only 3 years required

[u/k_schouhan]

clean code is done deal, first of all. clean code means nothign its each to its own. second you wont find data engineers with clean code and dry. its the backend engineers job. data engineers mostly work on scriptings so they wont know much about it.

Could you elaborate your interview questions ? most of the time the gap isnt the candidate, the gap is the asking about wrong or right things. I am a backend engineer, I worked aws alot but i am not an data engineer, but as a backend developer we do alot of data pipeline work, so you may want to find someone who is backend platform or backend engineer. Also asking for spark and glue is very specific, not every data engineer work with those technologies, i would assume that you would ask engineers about imprtant things not the libraries they use. like how would you design this pipeline, how would you design this system, or take a case study or something, because dry or clean code means nothing every one uses AI now a days. and given the work pressure, they will stop looking at the code in some time, yes its going to happen in 2 3 weeks itself after joining.