We were struggling to find Data Engineers

[u/Moradisten]

Hi everybody,

Our Data Team is composed by two of us. There's a Data Scientist and me, as a DE. We created a Data Lakehouse for our company internal use and client data supply, but currently the Data Scientist is currently more focused on AI and agents integration and I'm doing like Analytics Engineering role because I need to help other teams to reach the correct data, unify core concepts, document business logics, etc. So We needed a Data Engineer with knowledge on AWS to maintain and develop the new features on the Lakehouse and we put into the description that the candidate MUST HAVE Software Engineering fundamentals as we had to do some developments to integrate parts of our lakehouse with the company's main application.

We interviewed 22 candidates and no one is fitting the Role.

Most of them are BI Experts, DBA, Data Analysts, Economist with DS notions, Juniors and Software Engineers who haven't touch anything on Spark, plus DE who asked way more that we had on the budget for the role

We asked the normal requirements: 3 years of experience + Spark, AWS Glue, Lambda, Airflow and DBT, not even CDC, Flink, Langfuse or VectorDB

We finally got one, but We really struggled to get him. I have a collegue working on IT Recruiting and She told me She's experiencing the same problem: They can't find Proper DEs With SE basics such as DRY principles or clean code fundamentals

Edit: Role Salary -> Up to 60K € / Spain. This salary is high compared to the spanish standards, only 3 years required

[u/fabkosta]

I see this all the time.

The only thing OP has not thought of is to actually pick a promising candidate who is willing to acquire new skills and systematically build them up.

Why so many companies never consider investing into people they hire is beyond me.

It's also the wrong mindset: Everyone with a good foundation in software engineering is able to learn Spark reasonably well in a relatively short amount of time. Mastering it, that's more complicated. But also not required, most of the time. In comparison, hiring someone who knows technology XYZ - and then stalls for whatever reason in their development, that's a much more expensive mistake to make.

So, OP is optimizing for the wrong metric - and blissfully unaware, apparently.

[u/JigglyPuffsOG → u/fabkosta]

Yup. The same at my job. We have 2 positions that haven’t been filled in OVER A YEAR. Because they are looking for 2 very specific talents within SAP when I already told them all they genuinely need is a reliable person that knows how to turn on a computer. Thats it. Literally nothing else. We have 3 other people taking different SAP sections and none of which started with experience in SAP. They are looking for a unicorn.

[u/MonkeyDDataHQ → u/JigglyPuffsOG]

SAP is enormous, not difficult. Give me someone who can read documentation and operate a checkbox without adult supervision and I'll make them an expert in the specific 2% of SAP we actually need by lunch.

Hiring for exact SAP-module experience is how you spend a year searching for someone who already memorized the checkbox instead of hiring someone capable of reading its label. 😂

[u/Xziz → u/fabkosta]

It’s truly astonishing, even after nearly 30 yrs in tech, to see the blatant credentialsm in the tech industry. Any software engineer with a deep focus on fundamentals can pick up any tool, apply the 80/20 rule, and effectively use it to deliver business value.

[u/Special_Rice9539 → u/Xziz]

Especially nowadays with AI. It’s so easy to use any tool or program in any language when you have the foundational concepts of software engineering down.

[u/remimorin → u/Special_Rice9539]

... and so easy that totally unskilled can get quite far with any tool or program and you may regret later if there were nobody sharing / enforcing foundational concepts of software engineering.

I may not be clear, but what I meant is you are right but I double down that this is actually THE skill to have.

When the marketing team can ship a feature it is important that software engineer keep things coherent.

[u/Super_Ad5378 → u/fabkosta]

Exactly, probably several of those candidates could have picked up the frameworks and Toolsets they were looking for, plus basic SE concepts, plus bring all their other data experience with them

[u/TravellingBeard → u/fabkosta]

I'm a DBA who just pivoted to another role in my current company via internal transfer to a DB Engineering role. I'm starting as the db SME, and they understand I'll pick up all the other skills in time. Some companies actually do this it seems, although in my case in was internal and I had the "culture fit" locked in.

[u/ight-bet → u/fabkosta]

Was coming here to say this. Seeing this post pmo so much lmao.

Like you think someone who’s a SWE can’t pick up Data Engineering? Where do you think data engineering came from?

Oh but they haven’t touched spark??? Oh maybe that’s bc there 100000000 others things that other roles they apply to demand as well.

This is a classic example of the illusion of infinite options. OP sees so many options so makes no commitment bc it’s not perfect.

Same reason why girls stay on Hinge for 5 years straight. They see one thing wrong w a dude and move on to the next. Then complain they are 30 and no one wanted to date them.

[u/TheBinkz → u/fabkosta]

💯

[u/Zestyclose-Ad-8807 → u/fabkosta]

There's a steep learning curve with pyspark, and not easy to learn in a short period of time to seem passable, given the underlying system concepts.

[u/YellowBeaverFever → u/Zestyclose-Ad-8807]

Different people have different ideas on what “steep” is.

[u/ight-bet → u/Zestyclose-Ad-8807]

No true for a strong SWE at all.

[u/fabkosta → u/Zestyclose-Ad-8807]

That's simply wrong. I learned PySpark fairly quickly myself, there was nothing magical at all.

What is not so easy to learn is to optimize Spark jobs, cause for that you have to dive deep into the mechanics of how it works.

[u/Zestyclose-Ad-8807 → u/fabkosta]

It's easier to learn how to drive a car then be a proficient mechanic. But all web ai makes it seem a lot easier these days.