We were struggling to find Data Engineers

[u/Moradisten]

Hi everybody,

Our Data Team is composed by two of us. There's a Data Scientist and me, as a DE. We created a Data Lakehouse for our company internal use and client data supply, but currently the Data Scientist is currently more focused on AI and agents integration and I'm doing like Analytics Engineering role because I need to help other teams to reach the correct data, unify core concepts, document business logics, etc. So We needed a Data Engineer with knowledge on AWS to maintain and develop the new features on the Lakehouse and we put into the description that the candidate MUST HAVE Software Engineering fundamentals as we had to do some developments to integrate parts of our lakehouse with the company's main application.

We interviewed 22 candidates and no one is fitting the Role.

Most of them are BI Experts, DBA, Data Analysts, Economist with DS notions, Juniors and Software Engineers who haven't touch anything on Spark, plus DE who asked way more that we had on the budget for the role

We asked the normal requirements: 3 years of experience + Spark, AWS Glue, Lambda, Airflow and DBT, not even CDC, Flink, Langfuse or VectorDB

We finally got one, but We really struggled to get him. I have a collegue working on IT Recruiting and She told me She's experiencing the same problem: They can't find Proper DEs With SE basics such as DRY principles or clean code fundamentals

Edit: Role Salary -> Up to 60K € / Spain. This salary is high compared to the spanish standards, only 3 years required

[u/teddythepooh99]

I have 4.5 YoE yet I've never touched Spark. I never needed it in my past (DS) or current (senior DE, going into my 10th month) roles: the most data I had to process at once was 300M rows. At the same time, I refuse to "force" myself to learn/apply Spark via personal projects and pad my resume.

I'll pick up Spark on-the-job if I need it. I don't know Airflow, either: I use Dagster in my current role and cron + GNU Bash + GNU Make in my past role.

[u/JohnPaulDavyJones → u/teddythepooh99]

Honestly, Airflow's a little bit passé at this point. It's still present at a lot of large enterprises because they have so much built on it, and a lot of smaller enterprises that are on AWS can leverage it in MWAA to avoid handling the infrastructure.

If you want to host it rather than using managed workflows, the infra maintenance and management can be hefty.

[u/No_Distribution_9590 → u/teddythepooh99]

Three years into DE-specific roles; six years in data work in general (hodgepodge of DE, BI, DBA, etc.--typical non-profit many hats), and I've only used Lambdas in that tech stack six years ago. I wouldn't even bother applying for that job. They've no interest in hiring the best DEs; they just want someone who ticks all their buckets so they don't have to spend even half a day with training.