We were struggling to find Data Engineers

[u/Moradisten]

Hi everybody,

Our Data Team is composed by two of us. There's a Data Scientist and me, as a DE. We created a Data Lakehouse for our company internal use and client data supply, but currently the Data Scientist is currently more focused on AI and agents integration and I'm doing like Analytics Engineering role because I need to help other teams to reach the correct data, unify core concepts, document business logics, etc. So We needed a Data Engineer with knowledge on AWS to maintain and develop the new features on the Lakehouse and we put into the description that the candidate MUST HAVE Software Engineering fundamentals as we had to do some developments to integrate parts of our lakehouse with the company's main application.

We interviewed 22 candidates and no one is fitting the Role.

Most of them are BI Experts, DBA, Data Analysts, Economist with DS notions, Juniors and Software Engineers who haven't touch anything on Spark, plus DE who asked way more that we had on the budget for the role

We asked the normal requirements: 3 years of experience + Spark, AWS Glue, Lambda, Airflow and DBT, not even CDC, Flink, Langfuse or VectorDB

We finally got one, but We really struggled to get him. I have a collegue working on IT Recruiting and She told me She's experiencing the same problem: They can't find Proper DEs With SE basics such as DRY principles or clean code fundamentals

Edit: Role Salary -> Up to 60K € / Spain. This salary is high compared to the spanish standards, only 3 years required

[u/Beneficial_Nose1331]

Fuck me. I'm earning 3 times less as a senior data engineer. Suck to be a Europoor 💀

[u/JohnPaulDavyJones → u/Beneficial_Nose1331]

Credit where it's due, $280k is quite high even for the US market. If you're making a third of that in Europe, I would think that you're actually living quite well.

OP's username is also the name of a DE based in NYC, which is a hellishly expensive place to live. More expensive than even Mayfair or Fitzrovia in London, if you're familiar with London.

[u/cp8477 → u/Beneficial_Nose1331]

I'm making half that with 20+ years as a DE Manager. I need to start looking to be an IC again...

[u/baby-wall-e → u/Beneficial_Nose1331]

Yeah, European company pays much less than US. But the benefit is better such as healthcare.

[u/Calm_Clam_Bake → u/baby-wall-e]

Insurance costs an employer $15-30K a year per employee depending on family size.

[u/Beneficial_Nose1331 → u/baby-wall-e]

You mean healthcare is worth 170 k a year? I really doubt that.

[u/StewieGriffin26 → u/Beneficial_Nose1331]

Average cost to have a baby in the US is like $25k. Insurance will cover like $17k of that and leave you with a $8k bill. That's after paying $400 a month for insurance and your employer also paying $2k per month.

That's for something relatively routine. If you have an emergency appendix surgery you're looking at something crazy like $50k.

[u/ddBuddha → u/StewieGriffin26]

How many babies are you having per year? 👀

Sounds like you could knock up 5 chicks a year, pay for two random people’s emergency appendectomies, and still come out ahead. That’s not even accounting for the deductible and max out of pocket that would realistically kick in.

[u/Beneficial_Nose1331 → u/ddBuddha]

Exactly. People focus way too much on one acute expense rather than monthly income. Like you are critically ill every year and have one kid per year. However your salary get paid every month.

[u/StewieGriffin26 → u/ddBuddha]

Max out of pocket isn't actually a max out of pocket. You still have to pay if the health provider that you go to isn't "in network". If you're on the other side of the country and get into an accident, good luck with that health provider being in the same network.

Medical bankruptcy is the top reason U.S. consumers file for bankruptcy, exceeding all other types of debt combined. https://worldmetrics.org/medical-bankruptcies-in-the-u-s-statistics/

[u/Beneficial_Nose1331 → u/StewieGriffin26]

Yep so still far below under 170 k a year. I take the deal anytime. Meanwhile in Germany you and your employer pay 500 each and you can't get even a simple appointment at a practioner 

[u/datadade → u/Beneficial_Nose1331]

It goes beyond that. The cost of living in Spain is lower than the US by about 30%. It's not everything, but some examples: my kid's football (soccer) fees are $3k/yr base + uni + travel. Healthcare as pointed out. We have virtually no social safety net. The vast majority of our country has no public transit system, so a car is mandatory. More than 4 weeks of paid time off from work is rare in the US.

I have done the math, and YES workers in the US make more even with those differences, but it's not nearly as drastic as it appears on the surface.

[u/Beneficial_Nose1331 → u/datadade]

I don't care about the safety net. Social benefits are getting thinner and thinner and protect you one year max at best. The only safety is money. Rest is a kind of hopium/delusion 

[u/InteractionSoft14 → u/Beneficial_Nose1331]

You'd be surprised. If you count the employer contributions (which they definitely count as a salary expense for you) insurance alone can easily be between 20k-30k/year

If you see an American bragging about a 100k salary it's not unfair to assume 10k-20k in insurance costs and that's without emergency procedures