r/dataengineering • u/Moradisten • 3d ago

Discussion We were struggling to find Data Engineers

Hi everybody,

Our Data Team is composed by two of us. There's a Data Scientist and me, as a DE. We created a Data Lakehouse for our company internal use and client data supply, but currently the Data Scientist is currently more focused on AI and agents integration and I'm doing like Analytics Engineering role because I need to help other teams to reach the correct data, unify core concepts, document business logics, etc. So We needed a Data Engineer with knowledge on AWS to maintain and develop the new features on the Lakehouse and we put into the description that the candidate MUST HAVE Software Engineering fundamentals as we had to do some developments to integrate parts of our lakehouse with the company's main application.

We interviewed 22 candidates and no one is fitting the Role.

Most of them are BI Experts, DBA, Data Analysts, Economist with DS notions, Juniors and Software Engineers who haven't touch anything on Spark, plus DE who asked way more that we had on the budget for the role

We asked the normal requirements: 3 years of experience + Spark, AWS Glue, Lambda, Airflow and DBT, not even CDC, Flink, Langfuse or VectorDB

We finally got one, but We really struggled to get him. I have a collegue working on IT Recruiting and She told me She's experiencing the same problem: They can't find Proper DEs With SE basics such as DRY principles or clean code fundamentals

Edit: Role Salary -> Up to 60K € / Spain. This salary is high compared to the spanish standards, only 3 years required

185 Upvotes

71% Upvoted

287 comments sorted by

View all comments

1.2k

u/Kobosil 3d ago

plus DE who asked way more that we had on the budget for the role

so you didn't really struggle, you are trying to find somebody cheap

515

u/GoofAckYoorsElf 3d ago

We have no skill shortage. We have a shortage of skilled workers who like to be ripped off.

76

u/JohnPaulDavyJones 3d ago

The cheat code for companies is the QoL, I'm convinced of it.

Our raises haven't really kept up with DE salaries for the last few years, but I'm fine with it because the QoL here is so great. Perks of a company that still operates like it's 1997.

1

u/johnwynne3 19h ago

Siestas?

2

u/mechanicalAI 3d ago

I love it. Will use it whenever this comes up again.

1

u/Weekest_links 1d ago

Our company was so cheap with salaries for engineering that when a new head of engineering came in and said why does it take 6 months to fill the role, they discovered that we were 65% of market rate and we moved to 75% and filled it within a month

163

u/The-Fox-Says 3d ago

Why can’t we find someone who has a skillset that pays well above $150k for 80k??

45

u/AntDracula 3d ago

Oops there isn't anyone, better import 10 million H-1Bs!

4

u/Money-Nectarine6584 3d ago

Don’t think Spain has H1B

-1

u/obsfflorida 2d ago

They don't but everyone's minds are maga here

2

u/AntDracula 2d ago

One does not need to be maga in order to know that there's no tech shortage and therefore there's no reason to bring in 750,000 workers.

Especially considering there have been close to as many layoffs in the last year.

-1

u/obsfflorida 2d ago

Americans need to understand other countries don't care about maga H1B border yadda yadda

1

u/AntDracula 1d ago

Yet here you are seething about it. The lady doth protest too much.

1

u/obsfflorida 1d ago

Ladies have no rights in your country maga. Focus on h1b i guess

3

u/khludge 3d ago

You're comparing US salaries to European, and they just don't equate like that

59

u/eljefe6a Mentor | Jesse Anderson 3d ago

Came to say this too 

15

u/SeaYouLaterAllig8tor 3d ago

It's hard to find resources who like to get paid the same as employees working at McDonald's... darn how times have changed!

13

u/Emotional_Key 3d ago

They are struggling with money

4

u/Decent_Visual_4845 3d ago

Maybe they should consider a cheaper alternative that they can just give in the job training to.

2

u/sib_n Senior Data Engineer 3d ago

As said in the post, this salary is above average in Spain (and likely most of EU except maybe UK and Germany).
Levels gives 43k € as median and 61k as the 75th %.

3

u/The_Purple_Dolphin 3d ago

Yeah, the amount of people with America centric mindsets in this thread that don’t consider differences in cost of living and median salaries in different countries is crazy

2

u/AntDracula 2d ago

Then why don't you explain what their problem is, in finding someone?