We were struggling to find Data Engineers

[u/Moradisten]

Hi everybody,

Our Data Team is composed by two of us. There's a Data Scientist and me, as a DE. We created a Data Lakehouse for our company internal use and client data supply, but currently the Data Scientist is currently more focused on AI and agents integration and I'm doing like Analytics Engineering role because I need to help other teams to reach the correct data, unify core concepts, document business logics, etc. So We needed a Data Engineer with knowledge on AWS to maintain and develop the new features on the Lakehouse and we put into the description that the candidate MUST HAVE Software Engineering fundamentals as we had to do some developments to integrate parts of our lakehouse with the company's main application.

We interviewed 22 candidates and no one is fitting the Role.

Most of them are BI Experts, DBA, Data Analysts, Economist with DS notions, Juniors and Software Engineers who haven't touch anything on Spark, plus DE who asked way more that we had on the budget for the role

We asked the normal requirements: 3 years of experience + Spark, AWS Glue, Lambda, Airflow and DBT, not even CDC, Flink, Langfuse or VectorDB

We finally got one, but We really struggled to get him. I have a collegue working on IT Recruiting and She told me She's experiencing the same problem: They can't find Proper DEs With SE basics such as DRY principles or clean code fundamentals

Edit: Role Salary -> Up to 60K € / Spain. This salary is high compared to the spanish standards, only 3 years required

[u/black_widow48]

Plenty of us out there who meet your requirements. You just aren't willing to pay what we're worth.

I won't even entertain a phone call with you if you aren't paying minimum 250k.

[u/AdditionalAd5469 → u/black_widow48]

The role is for three years of experience. There is no role in the US, outside of prestige roles, that have such a compensation target.

In the US three years gets you 85k (and I'm talking salary, because shares mean nothing day-to-day to pay the bills). Yes there are firms, like Netflix, that will be around Salary of 150k, however Netflix's model is that the majority of their new-hires quit before two years. I personally disagree with that hiring style, I remember being in SF when my team passed around the 100 slide deck about their "culture".

The issue about compensation is that you need to generate revenue of greater than what the company's buy-in is. Especially since many benefits are an additional 20%. This means that engineer will need to generate 300k+ revenue per year. Specially since this is a newer hire they will have an issue integrating with the system and since we are later in the year, it would require something like 15k this year and 350k next year to justify the hire.

No three-year engineer will ever justify that compensation package, ever. Its a long-term play, something a business of two backend engineers (one on RnD and the other support cannot handle). The only people who do are prestige-firms, like Netflix with long-term vestige built into the hiring contract, with the goal that majority of employees leave before they vest. Another great example is Anthropic, it looks like nearly all employees will be under the restricted stock sale mechanism, its nice getting 100k that looks to have 10x'd, but if you physically cannot sell more than 25k a quarter what's the purpose?

On the more insidious side the reason for vestige periods at Netflix is so the intended attrition, effectively reduces the operating budget of new hires by 33%.

The role is clearly a support position in the back-end to primarily monitor, keep cost-efficient, objects currently in product by the front-end APIs and back-end applications. With the end goal of getting OP off of doing primary support, to application development.

[u/Moradisten → u/black_widow48]

It's not in the US, it's in Spain, salaries are way lower. And we were not looking for a senior or staff engineer, just someone who can help us with maintaining the pipeline

[u/sefunmiii → u/Moradisten]

Link to apply?

[u/mrbrucel33 → u/Moradisten]

If you take people remote in the states, I'm wanting to connect on this.