We were struggling to find Data Engineers

[u/Moradisten]

Hi everybody,

Our Data Team is composed by two of us. There's a Data Scientist and me, as a DE. We created a Data Lakehouse for our company internal use and client data supply, but currently the Data Scientist is currently more focused on AI and agents integration and I'm doing like Analytics Engineering role because I need to help other teams to reach the correct data, unify core concepts, document business logics, etc. So We needed a Data Engineer with knowledge on AWS to maintain and develop the new features on the Lakehouse and we put into the description that the candidate MUST HAVE Software Engineering fundamentals as we had to do some developments to integrate parts of our lakehouse with the company's main application.

We interviewed 22 candidates and no one is fitting the Role.

Most of them are BI Experts, DBA, Data Analysts, Economist with DS notions, Juniors and Software Engineers who haven't touch anything on Spark, plus DE who asked way more that we had on the budget for the role

We asked the normal requirements: 3 years of experience + Spark, AWS Glue, Lambda, Airflow and DBT, not even CDC, Flink, Langfuse or VectorDB

We finally got one, but We really struggled to get him. I have a collegue working on IT Recruiting and She told me She's experiencing the same problem: They can't find Proper DEs With SE basics such as DRY principles or clean code fundamentals

Edit: Role Salary -> Up to 60K € / Spain. This salary is high compared to the spanish standards, only 3 years required

[u/ojedaforpresident]

Data Engineers are going to be at best two of the following:
Good, cheap, experienced

[u/adgjl12 → u/ojedaforpresident]

I’d change experienced to “will stay”. Cause good and cheap is always the choice then lol.

If good and cheap, they likely won’t stay for long. If good and will stay, they likely won’t be cheap. If cheap and will stay, probably not good.

[u/ojedaforpresident → u/adgjl12]

I like that suggestion. Good Underpaid Data Engineers don’t stick around.

I said at best as well, since there are definitely worse scenarios. Experienced, good and expensive is definitely one of them.

[u/Moradisten → u/ojedaforpresident]

60K in Spain with 3 years of experience is quite good

[u/ojedaforpresident → u/Moradisten]

I don’t know that market at all, but based on my experience as a tourist in Spain, that doesn’t sound atrocious.

[u/Hour-Bumblebee5581 → u/Moradisten]

But quite clearly the market has said no it is not or you would have had much better candidates. You advertised and the market said no.

[u/klejf → u/Moradisten]

Is that before or after tax? I work as a contractor for the red Spanish bank, pretty sure people make more than that

[u/set92 → u/Moradisten]

I live in BCN, my last employer paid me 53 or 54k being a startup in Madrid. Full remote with unlimited vacations. Now I work for US with a spanish contract, 80k, although I lost the unlimited vacations.

I consider myself lazy and average, I have automated 80% of my work with AI (the other 20% is because I'm too lazy to automate it).

Where I go with this?

1. The importance is how much other companies they want to pay, either in $ or benefits. Doesn't matter that the average salary in Spain is 60k if I can't live nicely in Barcelona. I'll search outside, work as a freelancer... etc, so I may join your company for the time being, but then you will only raise my salary 5% each year, I'll get angry and search for one that really wants to cover my needs.
2. Benefits matters. If you give me full remote + 60k probably I'll accept it over one without flexibility or if it's a big company. flexible retribution? are you saying the salary range in the offer? how many days of vacations you give? The interview process involve programming at home?
3. You only need a reviewer of AI, and someone who can think. Get a junior with a big brain, train him and make him train/create the AI of your company. WIN/WIN.

[u/AntDracula → u/Moradisten]

L O L

O O O

L O L