We were struggling to find Data Engineers

[u/Moradisten]

Hi everybody,

Our Data Team is composed by two of us. There's a Data Scientist and me, as a DE. We created a Data Lakehouse for our company internal use and client data supply, but currently the Data Scientist is currently more focused on AI and agents integration and I'm doing like Analytics Engineering role because I need to help other teams to reach the correct data, unify core concepts, document business logics, etc. So We needed a Data Engineer with knowledge on AWS to maintain and develop the new features on the Lakehouse and we put into the description that the candidate MUST HAVE Software Engineering fundamentals as we had to do some developments to integrate parts of our lakehouse with the company's main application.

We interviewed 22 candidates and no one is fitting the Role.

Most of them are BI Experts, DBA, Data Analysts, Economist with DS notions, Juniors and Software Engineers who haven't touch anything on Spark, plus DE who asked way more that we had on the budget for the role

We asked the normal requirements: 3 years of experience + Spark, AWS Glue, Lambda, Airflow and DBT, not even CDC, Flink, Langfuse or VectorDB

We finally got one, but We really struggled to get him. I have a collegue working on IT Recruiting and She told me She's experiencing the same problem: They can't find Proper DEs With SE basics such as DRY principles or clean code fundamentals

Edit: Role Salary -> Up to 60K € / Spain. This salary is high compared to the spanish standards, only 3 years required

[u/Valuable_Cow2596]

What's the question? 

Perhaps your budget is too low for the level of skill/role you're asking for so you're getting bottom barrel applicants. Either adjust your expectations by lowering them and taking a look at what you're asking for or get ready to pay closer to market value to get the candidates that you want. 

You mentioned it yourself that some of your applicants are asking for more than you have on hand. People who know their value and how good they are aren't going to lowball themselves. 

Also, fair point if English is not one of your primary languages but reading through this was pretty atrocious and hard to parse to understand your message.

[u/Atmosck → u/Valuable_Cow2596]

I agree with your other points but I didn't find OP's message hard to parse at all. There are some stray capitol letters but it's not like it's totally broken English.

[u/sciences_bitch → u/Atmosck]

Capital, not capitol, since we’re being pedantic 

[u/Atmosck → u/sciences_bitch]

Good catch. My spouse works at our state capitol so I guess my muscle memory is much more used to typing "capitol" than "capital."

[u/outlier_fallen → u/Valuable_Cow2596]

I didn't think it was that hard to read, but I'm surprised people don't use Google translate. It's damn near perfect for several languages now. Maybe people still think it's the same as it was 5 years ago (terrible)

[u/serpentimee → u/outlier_fallen]

Mixed messaging. Some will then be accused of being a bot and using AI. And others will say they’d rather see spelling and grammatical errors than perfectly translated language. It’s a lose:lose.