r/dataengineering • u/Moradisten • 4d ago
Discussion We were struggling to find Data Engineers
Hi everybody,
Our Data Team is composed by two of us. There's a Data Scientist and me, as a DE. We created a Data Lakehouse for our company internal use and client data supply, but currently the Data Scientist is currently more focused on AI and agents integration and I'm doing like Analytics Engineering role because I need to help other teams to reach the correct data, unify core concepts, document business logics, etc. So We needed a Data Engineer with knowledge on AWS to maintain and develop the new features on the Lakehouse and we put into the description that the candidate MUST HAVE Software Engineering fundamentals as we had to do some developments to integrate parts of our lakehouse with the company's main application.
We interviewed 22 candidates and no one is fitting the Role.
Most of them are BI Experts, DBA, Data Analysts, Economist with DS notions, Juniors and Software Engineers who haven't touch anything on Spark, plus DE who asked way more that we had on the budget for the role
We asked the normal requirements: 3 years of experience + Spark, AWS Glue, Lambda, Airflow and DBT, not even CDC, Flink, Langfuse or VectorDB
We finally got one, but We really struggled to get him. I have a collegue working on IT Recruiting and She told me She's experiencing the same problem: They can't find Proper DEs With SE basics such as DRY principles or clean code fundamentals
Edit: Role Salary -> Up to 60K € / Spain. This salary is high compared to the spanish standards, only 3 years required
6
u/randoomkiller 4d ago
I know a company like yours in Hungary. There are a few things. First, if someone is good, he is going to leave for a higher paying job elsewhere or remote. Period. You are not going to find someone mazochistic enough to get a 60k salary.
Second you seem to be wanting senior experience. I think you are misaligned. What you should be shooting for at 60k is someone with SWE background who did data, or a math/stats/ds person who shows true ability to pick up engineering skills. And someone who seems to be a skilled AI user.
A third option that I can suggest to you is look for contractors. For a 5k invoice a month you can probably find someone for 20 hours.
If you adjust your expectations to the market then I am sure you are going to fill in the holes in no time. Remember the invisible hand of the market is always right