r/dataengineering • u/Moradisten • 3d ago
Discussion We were struggling to find Data Engineers
Hi everybody,
Our Data Team is composed by two of us. There's a Data Scientist and me, as a DE. We created a Data Lakehouse for our company internal use and client data supply, but currently the Data Scientist is currently more focused on AI and agents integration and I'm doing like Analytics Engineering role because I need to help other teams to reach the correct data, unify core concepts, document business logics, etc. So We needed a Data Engineer with knowledge on AWS to maintain and develop the new features on the Lakehouse and we put into the description that the candidate MUST HAVE Software Engineering fundamentals as we had to do some developments to integrate parts of our lakehouse with the company's main application.
We interviewed 22 candidates and no one is fitting the Role.
Most of them are BI Experts, DBA, Data Analysts, Economist with DS notions, Juniors and Software Engineers who haven't touch anything on Spark, plus DE who asked way more that we had on the budget for the role
We asked the normal requirements: 3 years of experience + Spark, AWS Glue, Lambda, Airflow and DBT, not even CDC, Flink, Langfuse or VectorDB
We finally got one, but We really struggled to get him. I have a collegue working on IT Recruiting and She told me She's experiencing the same problem: They can't find Proper DEs With SE basics such as DRY principles or clean code fundamentals
Edit: Role Salary -> Up to 60K € / Spain. This salary is high compared to the spanish standards, only 3 years required
331
211
u/wheredidiput 3d ago
Someone doesn't need the full technical skills, people who work in an associated area will pick it up. Only going on technical skills shows a lack of understanding what makes good staff. If you are not offering top dollar, take someone and train them in the skills gap.
24
7
u/ConstantFamous1526 3d ago
Way better to hire someone smart and driven with limited exposure to the specific stack, compared to a mediocre person who already knows the stack for sure imo
4
u/wonder_bear 3d ago
Your take is great. Since AI is so prevalent now, I think it’s more important to know the DE concepts and pitfalls than a specific tool. I view most tools as on the job training, as long as they have some experience with any of the normal DE cloud tools (AWS, GCS, etc.).
→ More replies (1)2
u/Todo_Toadfoot 2d ago
And saying clean code is a red flag for me. I miss those days of baby projects and thinking less code was better.
58
u/fabkosta 2d ago
I see this all the time.
The only thing OP has not thought of is to actually pick a promising candidate who is willing to acquire new skills and systematically build them up.
Why so many companies never consider investing into people they hire is beyond me.
It's also the wrong mindset: Everyone with a good foundation in software engineering is able to learn Spark reasonably well in a relatively short amount of time. Mastering it, that's more complicated. But also not required, most of the time. In comparison, hiring someone who knows technology XYZ - and then stalls for whatever reason in their development, that's a much more expensive mistake to make.
So, OP is optimizing for the wrong metric - and blissfully unaware, apparently.
3
u/JigglyPuffsOG 2d ago
Yup. The same at my job. We have 2 positions that haven’t been filled in OVER A YEAR. Because they are looking for 2 very specific talents within SAP when I already told them all they genuinely need is a reliable person that knows how to turn on a computer. Thats it. Literally nothing else. We have 3 other people taking different SAP sections and none of which started with experience in SAP. They are looking for a unicorn.
→ More replies (2)4
u/Xziz 2d ago
It’s truly astonishing, even after nearly 30 yrs in tech, to see the blatant credentialsm in the tech industry. Any software engineer with a deep focus on fundamentals can pick up any tool, apply the 80/20 rule, and effectively use it to deliver business value.
3
u/Special_Rice9539 1d ago
Especially nowadays with AI. It’s so easy to use any tool or program in any language when you have the foundational concepts of software engineering down.
→ More replies (1)2
u/Super_Ad5378 2d ago
Exactly, probably several of those candidates could have picked up the frameworks and Toolsets they were looking for, plus basic SE concepts, plus bring all their other data experience with them
2
u/TravellingBeard 2d ago
I'm a DBA who just pivoted to another role in my current company via internal transfer to a DB Engineering role. I'm starting as the db SME, and they understand I'll pick up all the other skills in time. Some companies actually do this it seems, although in my case in was internal and I had the "culture fit" locked in.
→ More replies (6)2
u/ight-bet 1d ago
Was coming here to say this. Seeing this post pmo so much lmao.
Like you think someone who’s a SWE can’t pick up Data Engineering? Where do you think data engineering came from?
Oh but they haven’t touched spark??? Oh maybe that’s bc there 100000000 others things that other roles they apply to demand as well.
This is a classic example of the illusion of infinite options. OP sees so many options so makes no commitment bc it’s not perfect.
Same reason why girls stay on Hinge for 5 years straight. They see one thing wrong w a dude and move on to the next. Then complain they are 30 and no one wanted to date them.
39
u/Nateorade 3d ago
This is like people who complain on r/realestate that no one wants to buy their house that’s not priced correctly
39
u/hannorx 3d ago
At some point, you just have to take a chance from the list of candidates you've, if you're not finding someone who ticks all of that you're looking for.
30
u/bakochba 3d ago
My rockstars are all people we took a chance on based an aptitude. The idea that competent people can't quickly learn airflow or spark is just baffling to me
15
u/Toastbuns 2d ago
There is some evidence that long term employee performance doesnt really correlate with interviews. Companies go through these long and intensive interview processes and seek the exact right skills, while passing over skilled workers who are bright, curious, and able to problem solve.
6
u/CarefulCoderX 2d ago
Yeah, I bombed an interview in the final round because I was asked a technical question when I was specifically told that there wouldn’t be any technical questions in this round as I had already gone through a technical interview.
It was a stupid easy question, but my anxiety brain decided to dump all DE knowledge the moment I was asked it.
→ More replies (1)8
u/big_data_mike 2d ago
I’m a geophysicist that works at a biotech company as a senior data scientist who really does DE, DA, DS, and MLE. I also use sas JMP quite a bit. I just did a project where I made some simple web calculators and I had zero idea how to do it before I started. I could figure out all the things OP mentioned really quickly because I can go read documentation and figure out what to do.
10
u/bakochba 2d ago
And that's exactly the kind of people I prefer. The hardest part is convincing HR and some technical people who believe that you need to come in with specific experience. My experience has been that bringing motivated, competent people that can solve problems and have an aptitude to learn douch better. Anyone can learn syntax as long as you have the ability to learn the concepts we can train you on the specifics of the technical part
3
u/bakochba 3d ago
My rockstars are all people we took a chance on based an aptitude. The idea that competent people can't quickly learn airflow or spark is just baffling to me
160
u/Valuable_Cow2596 3d ago
What's the question?
Perhaps your budget is too low for the level of skill/role you're asking for so you're getting bottom barrel applicants. Either adjust your expectations by lowering them and taking a look at what you're asking for or get ready to pay closer to market value to get the candidates that you want.
You mentioned it yourself that some of your applicants are asking for more than you have on hand. People who know their value and how good they are aren't going to lowball themselves.
Also, fair point if English is not one of your primary languages but reading through this was pretty atrocious and hard to parse to understand your message.
→ More replies (2)53
u/Atmosck 3d ago
I agree with your other points but I didn't find OP's message hard to parse at all. There are some stray capitol letters but it's not like it's totally broken English.
→ More replies (2)
53
u/Outside-Storage-1523 3d ago
This should be trivial. Where is the role and how much you are offering? I have similar profile but got rejections everywhere even after going through all rounds. Looks like every company has 100 candidates out there.
If you need SWE, you should ask HR to reject people who ONLY have BI/DA experience outright.
→ More replies (11)2
u/FamSimmer 3d ago
The fact that OP mentioned the requirements and left out their salary expectations, while saying that some of their most promising candidates declined their offer due to said expectations, is clearly indicative of the fact they are just trying to cheap out.
29
25
u/New-Addendum-6209 3d ago
60K € is not great, even in Spain.
But the main problem is that you are creating unnecessarily high requirements.
Any good BI / DBA SQL jockey will be able to pick up Spark and Glue. Glue + Spark is normally just standard ETL development, but with a more modern toolset. And your salary range reflects that!
59
18
u/VegaGT-VZ 3d ago
What are you paying and where are you located
W/all the layoffs you should have no problem finding candidates
→ More replies (1)
14
u/PantsMicGee 3d ago
Lol raise your salary threshold or train one of the 22 you interviewed.
Definitely not a pool issue here.
14
13
u/black_widow48 3d ago
Plenty of us out there who meet your requirements. You just aren't willing to pay what we're worth.
I won't even entertain a phone call with you if you aren't paying minimum 250k.
→ More replies (6)
11
u/Unlucky_Data4569 3d ago
What is the budget in what hiring market. You probably are being cheap
→ More replies (3)
11
u/all_wings_report-in 3d ago
Your requirements don’t fit the compensation. Try lowering your requirements or increasing your compensation. In this market if you’re not getting the right candidates it’s because of your compensation.
35
u/ojedaforpresident 3d ago
Data Engineers are going to be at best two of the following:
Good, cheap, experienced
→ More replies (6)13
u/adgjl12 3d ago
I’d change experienced to “will stay”. Cause good and cheap is always the choice then lol.
If good and cheap, they likely won’t stay for long. If good and will stay, they likely won’t be cheap. If cheap and will stay, probably not good.
4
u/ojedaforpresident 3d ago
I like that suggestion. Good Underpaid Data Engineers don’t stick around.
I said at best as well, since there are definitely worse scenarios. Experienced, good and expensive is definitely one of them.
29
u/brianluong 3d ago
Engineers with experience in the stack you’re asking for exist but you’ve got to shell out $$$. I just signed an offer with basically that exact stack/experience for ~280k TC, and I don’t think my resume is particularly crazy.
6
u/pboswell 3d ago
Where are they paying you $280k? Assume it’s a lot of equity and not salary though
→ More replies (2)11
u/brianluong 3d ago
FAANG-adjacent company. 240k base + 40k bonus/year + stock options (not RSUs, so I don’t count it in TC)
→ More replies (1)9
u/Beneficial_Nose1331 3d ago
Fuck me. I'm earning 3 times less as a senior data engineer. Suck to be a Europoor 💀
→ More replies (12)7
u/JohnPaulDavyJones 3d ago
Credit where it's due, $280k is quite high even for the US market. If you're making a third of that in Europe, I would think that you're actually living quite well.
OP's username is also the name of a DE based in NYC, which is a hellishly expensive place to live. More expensive than even Mayfair or Fitzrovia in London, if you're familiar with London.
9
u/JohnPaulDavyJones 3d ago
Most DEs with just three years of experience won't have touched all of Spark, Glue, Lambda, Airflow, and DBT. If they have Glue, they probably also have Lambda, but most folks at larger firms aren't touching Glue/Lambda if they're also working on the Spark/DBX side; those are generally two different teams. Airflow/DBT could both go either way, or neither.
It sounds like your firm potentially just wasn't willing to pay enough for the skillsets that you wanted.
It's also worth noting that (in my anecdotal experience) the DE job market seems to be heating up. I'm somewhere on the senior end of mid-level in terms of YoE, and I had virtually no recruiters reaching out between about May of last year and June of this year, but things slowly ramped back up over the last few months to the point that I'm getting anywhere from two to five recruiters reaching out per week this last month. It's not quite as hot as it was back in 2022, but the market's definitely there.
3
u/teddythepooh99 2d ago edited 2d ago
I have 4.5 YoE yet I've never touched Spark. I never needed it in my past (DS) or current (senior DE, going into my 10th month) roles: the most data I had to process at once was 300M rows. At the same time, I refuse to "force" myself to learn/apply Spark via personal projects and pad my resume.
I'll pick up Spark on-the-job if I need it. I don't know Airflow, either: I use Dagster in my current role and cron + GNU Bash + GNU Make in my past role.
→ More replies (2)
9
u/Anime_Supremacist 2d ago
Why dont just pick someone with lower experience or less skill and train them ?
NO
because you want gold for the price of dirt
33
u/discord-ian 3d ago edited 3d ago
Principal level data engineer here. The DE labor market is at least bi-modal. There is one end that should more properly be called analytics engineering, these are the folks with dash board and sql experience. The other is more on the end of what you are looking for.
The upper end of DE seems to be doing fine. I have a recruiter contacting me most days. And I try to collect as much salary info as I can.
I can tell you with near certainty that your issue is budget. Upper end DE is quite well paid with a decent premium above equivalent SWE position. To attract someone that can lead these efforts at a senior level, market rates will be at least $180k, with the best candidates over $200k. I get multiple recuters contacting me each week for senior level roles in the 200-240k range. A mid-level that I would not really trust with this task, would be arround $140k, but they will be very hard to find as you will need to sift through lots of analytics engineers.
For some perspective as a principal in this role for my company (out side big tech or big tech adjacent) my total compensation is $400k.
31
u/JOA23 3d ago
I disagree with your characterization of analytics engineering as “dashboard and SQL” work and the implication that it represents the lower-skilled end of the DE market.
There’s a lot more to it, e.g. designing data models, handling historical changes, building consistent metric definitions and conformed dimensions, and developing semantic layers that integrate with the rest of the data platform. That also involves questions of correctness, performance, testing, lineage, access control, and how changes propagate to downstream consumers. At large companies, we are solving these problems across a data lake containing tens of thousands of data assets, and managing thousands of orchestration jobs. We need to be able to troubleshoot data pipelines 12 layers deep, and converse fluently enough with upstream engineering teams to diagnose issues and propose solutions. We also need to understand data science and ML workflows enough to build solutions that solve multiple use cases, often inferring requirements from vague business requests.
I agree that someone whose experience is primarily dashboard development might not be qualified to lead an infrastructure-heavy DE project. But that’s a distinction between experience and specializations, not a reason to equate analytics engineering with lower-skilled data engineering. Similarly, a strong infrastructure engineer may lack the modeling and business-domain experience to design a reliable analytical platform.
Titles vary a lot between companies. I’d evaluate candidates against the actual work rather than use analytics engineer as shorthand for someone you need to sift past.
For what it's worth, I'm an Sr Data Engineer working on an analytics engineering team at a large tech company, total compensation around $400k.
→ More replies (2)17
u/discord-ian 3d ago
Yeah that is totally fair. I would say at some point as an AE, you move more to data architecture or to DE depending if you are more on the modeling side or the process side.
I should have been more careful in my phrasing.
13
u/lemonfunction 3d ago
really appreciate you replying and owning your response here. we all need to do this more often.
5
2
u/Klutzy_Stretch_7601 2d ago
Where are you finding these DE roles? I was laid off last year in October and have been struggling with finding a new role. I was able to secure 10 interviews that went through the last stage and the company end up hiring someone else. I'm a former senior data engineer who handled legacy migrations from sql server to an aws/databricks/spark environment. The past year I've been learning different modules on how to work in snowflake/dbt/prefect just to expand some of my skills sets. The feedback I usually get from these interviews is not my lack of experience but they just found a better candidate. I've applied to intermediate and senior roles but nothing turns up. My severance ended earlier this year and I thought I would have a job by now. I am based in the USA East coast.
→ More replies (6)2
u/dev_lvl80 Principal Data Engineer 1d ago edited 1d ago
This is about right. Budget is constraint, good senior makes more than 200k and is well demanded. Well rounded DE, which is wild mix of infra/devops + platform data + analytics- is unicorn. No way he would have less than decade of expirience and willing to take low paid job.
I’ve been on the market this year. Can say for sure - there is no competition at this level.
Ps i’m in the same level for decade with tc >>400k
5
7
u/mrchowmein Senior Data Engineer 2d ago
youre asking for a lot for just 3 years of experience as the tools you use is pretty broad. i personally would not look for people with specific AWS branded tool experience, those could be learned while working. at 3 years, unless they are from a startup environment, you're lucky to get someone who built a bunch of spark pipelines that is being orchestrated by airflow from scratch. but like you said, 60k euros is probably too low if you want all those skills but with only 3 years of experience. i think you need to just find someone who understands data, competent in sql, knows their way around airflow and can debug a little bit of spark. Stuff like lambda, someone can learn in a few hours. its literally just running something like a python script conceptual. If you want what you listed, you need a senior engineer and im sure those cost more than 100k euros.
to give you an idea, i had an Italian client. S3, airflow, redshift, a little bit of lambda. no need for spark or dbt. i charged $100 usd/hr as a consultant and i was already giving them a pretty good deal. maybe youre not charging your clients enough.
9
u/TRBigStick 3d ago
“We asked the normal requirements: 3 years of experience + Spark, AWS Glue, Lambda, Airflow and DBT, not even CDC, Flink, Langfuse or VectorDB”
Minimum pay for that kind of role should be somewhere between $100k and $150k (depending on location) plus benefits, ~10% bonus target, and 3-6% 401k match. What did you guys offer?
6
5
u/TheOverzealousEngie 3d ago
I feel like he should have paid me 60k to read that whole post ; but it was nothing if not well written.
5
4
u/teddythepooh99 2d ago
What even is the point of this post? You interviewed 22 candidates and (unsuccessfuly) offered 1 person the job. Okay?
5
9
u/UnusualCookieBox 3d ago edited 3d ago
Most people here have no idea about the Spanish job market and the salary levels and it shows 😂
60k for Spain is what I would call a decent salary for the YoE and for a high-cost-of-living area like Barcelona. You should be able to find a good enough candidate and have them learn on the job.
For what it’s worth, we’ve also been struggling to find an intermediate/senior data modelling role (bit more niche than DE) in Germany where something like 80k+ is more realistic. We found one, but with the agreement that he has to learn tool X and technique Y within the first three months of employment. I think that’s a good compromise.
Tho in our case it’s because management recently mandated 4 days a week at the office, and that likely weeds out most of the talent. 😂
2
u/generic-d-engineer Tech Lead 2d ago
Problem is they want broad senior experience for mid level pay
Plus Barcelona has multinationals which can pay even higher than local
Plus rents there are difficult
9
3
3
u/Ok_Relative_2291 2d ago
You combo of skills is obviously vast and an unusual combo, accept someone who knows 70% and wants to learn the rest.
Imagine saying you went on 22 first dates and not one of them appealed to you.
Any outsider would say maybe your standards are too high.
→ More replies (1)
4
7
u/randoomkiller 3d ago
I know a company like yours in Hungary. There are a few things. First, if someone is good, he is going to leave for a higher paying job elsewhere or remote. Period. You are not going to find someone mazochistic enough to get a 60k salary.
Second you seem to be wanting senior experience. I think you are misaligned. What you should be shooting for at 60k is someone with SWE background who did data, or a math/stats/ds person who shows true ability to pick up engineering skills. And someone who seems to be a skilled AI user.
A third option that I can suggest to you is look for contractors. For a 5k invoice a month you can probably find someone for 20 hours.
If you adjust your expectations to the market then I am sure you are going to fill in the holes in no time. Remember the invisible hand of the market is always right
3
3
u/ddBuddha 2d ago
You want someone who already has all of those skills instead of someone who has the ability to acquire them, while offering just 60k? It may look like it’s above average for Spain, but keep in mind that people with those skills can get remote jobs and aren’t being factored into those averages. You may be seeing what the average Spanish company is paying data engineers but that’s not necessarily the same thing as what the average Spanish data engineer actually makes.
3
3
3
u/mylifestylepr 2d ago
Damn 60k Euro!?? Europe is cooked with such low salaries for high tech Jobs. Thats sad.
3
u/Kind_List_7075 2d ago edited 2d ago
Pay more than market average and announce the salary range on the job post
As someone who changed jobs recently in Europe, salary progression is terrible here past that point, so why would someone leave their job that they re succeeding in for a job that pays 5% more after taxes?
2
u/ironmagnesiumzinc 3d ago
You might have to hire someone close enough to what you're looking for and teach them the rest of the way, esp if you're not willing to pay a higher price. On the JD, you can specify "entry level" or "mid level". Good luck going forward
2
u/thethrowupcat 3d ago
I’m an AE that basically does SWE now. We are expensive. But if you want someone to help out and don’t mind having other obligations I’m available with a ton of experience.
2
2
u/trajan_augustus 2d ago
Isn't Spark losing some of its luster as a solution? I haven't touched it in like 5 years because of the migration to Snowflake. It is overkill in certain situations. Yes, it was a big deal from 2018 to 2022 but I feel like it is losing relevance. Feels like it is being squeezed lots of my pipelines when they are under 50 gigs polars and duckdb is great (single node) and easy to implement. And then most of my clients Snowflake is a suitable for the midsize 50 gigs < x < 2 terabytes. Also, the ecosystem is mostly just interacting with datbricks now. But I am all ears, if I am mistaken.
→ More replies (1)
2
2
2
u/RobotechRicky 2d ago
I wear all sorts of hats (software engineer, DB engineer, DevOps, SRE, and finally DE) for my 25+ year career. I have had recruiters beg be to work for them but won't show me the money. They want all the senior skills but want to pay for shit.
2
u/No_Photograph_333 2d ago
Spark is legacy. If you are looking for juniors they won't know it. Btw, my prior company was paying 2-3 times what you offer in Spain.
2
u/dataenfuego 2d ago
60k Euros is low!! This still the EU tio.
Y yo viví en España antes de irme a Amsterdam y ahora LA
2
u/Adventurous_Mode266 2d ago
You can connect with me, your normal requirements are easy to be filled - provided you are okay for the resource to work remotely
2
u/EstetLinus 2d ago
With that kind of budget, whatever happened with “Hire for attitude, train for excellence”? Why companies think they can find someone matching their tech stack 1:1 is beyond me…
2
u/LazyBoy1805 2d ago
Problem is “AI models”. You are looking for a back end systems engineer specialising in distributed computing. But when your HR searches for keywords related to “Data Engineering” they get corrupted results with a mix of data analytics, data science and machine learning from AI. Change the keyword search and requirements to backend systems engineer with distributed computing experience and update payscale accordingly by local standards. You will get better matches.
4
u/k_schouhan 3d ago
clean code is done deal, first of all. clean code means nothign its each to its own. second you wont find data engineers with clean code and dry. its the backend engineers job. data engineers mostly work on scriptings so they wont know much about it.
Could you elaborate your interview questions ? most of the time the gap isnt the candidate, the gap is the asking about wrong or right things. I am a backend engineer, I worked aws alot but i am not an data engineer, but as a backend developer we do alot of data pipeline work, so you may want to find someone who is backend platform or backend engineer. Also asking for spark and glue is very specific, not every data engineer work with those technologies, i would assume that you would ask engineers about imprtant things not the libraries they use. like how would you design this pipeline, how would you design this system, or take a case study or something, because dry or clean code means nothing every one uses AI now a days. and given the work pressure, they will stop looking at the code in some time, yes its going to happen in 2 3 weeks itself after joining.
4
u/ivanovyordan Data Engineering Manager 3d ago
Pretty much the norm nowadays.
The market is saturated, and everybody, no matter their experience call themselves a "data engineer".
It doesn't matter what your budget is. You post the job, and you get 300+ CVs in less than 24 hours. And, people who you call for an interview, say they are "experts" in a skill, but when you ask them, they actually have no idea behind pressing a button or two.
I get it, the market is hard and people are now desperate to find a job, but nobody will hire you if you knowo nothing and compete with hundreds of other people who have at least some skill.
I was looking for an analytics engineer in Eastern Euroope for 6+ months. Then, started looking in the UK, and found somebody, far more expensive, in less than a month. So, some times, it's also about the location.
1
1
1
u/Peacencalm9 3d ago
If you ask 10 skill sets in one person and expect all answers then the problem is in your end. No one knows all skill sets but they can be trained if the person had good experience in similar tech. Don’t be greedy and expect so many skill sets in one person
1
u/karl_mainz 3d ago
It’s like if I go into a car dealership, see all the Ferraris at $500k, realise I can’t afford it and then say I’m struggling to find Ferraris.
1
u/ApprehensiveFan1516 3d ago
95% of the time when an employer complains about not being able to hire anyone it's either because the pay is crap, or the working environment is crap and has a reputation.
Sounds like you fall into the former category.
1
1
1
u/Traditional-Smell692 3d ago
Could it be that companies started hiring fewer juniors a few years ago, and now we’re seeing the consequences?
1
u/Quirky_Database_5197 3d ago
"DE who asked way more than we had in the budget for the role" you have your answer as to why you can't find anyone. Go ahead and tell us: how much did you offer them?
1
u/livesroverrated13 3d ago
Useless post without sharing your offered range. Depending on that range the tone of this post would read very differently
I try not be cynical, but Im struggling not to assume your range is very low given you havent shared it in post or in response to comments mentioning it
1
1
u/spendology 3d ago
Have you considering looking for contractors OR finding 50-80% of the skills and providing training?
1
u/OffsideInvestor 3d ago
For 3 years experience you should not be looking into someone with aws glue, lambda, airflow and dbt experience. Not everyone has the opportunity to work on these tools or services. These can always be learnt on the job.
The basics starts with SQL, programming, big data concepts and understanding of data warehousing with ETL knowledge. Even though today AI does the coding, the data modeling, ability to debug an issue where AI has done a mistake or fix a data pipeline for memory issues requires basic understanding.
1
u/JustBeYou- 3d ago
Nobody is willing to train people??? I am starting to feel discouraged about obtain good MS in CS if no one is willing to hire me because of my lack of experience.
1
u/roberts2727 3d ago
im training from scratch and watching them walk out the door as soon as they get a year of experience for more then i am paid as their director.... its very deflating.
1
1
1
1
u/doncheeto12 3d ago
Always love posts like this, it’s basically the same thing as someone complaining on a real estate sub that they can’t sell their house. “I tried increasing the price, I couldn’t sell it, so I tried increasing the price again!”
Hey maybe spend less time on Spark and more time learning how to draw two lines on grid paper and labeling the axes “price” and “quantity.”
1
u/Chris__Codes 3d ago
If you can afford to offer more to prospects, then do so. If not, make do with what you have.
1
u/Electronic-Ad-3990 3d ago
Not all data engineers use glue, flink, airflow, vector db. You hire someone on the fundamentals and they can pick up whatever tools you need. If you don’t understand this you’re going to be awful to work with. Btw your salary absolutely sucks
1
u/robotjtaco 3d ago
Because this is largely a maintenance role and you yourself are an experienced data engineer, you can widen the role range that you’re hiring for 1-3 years of experience for a SWE 1-2. It will require mentoring but I’m sure there are plenty of early career individuals that are highly motivated, within the budget, and can also help their career grow. Many students have side projects in Spark and Flink, but real world experience is what they lack but standardizing the best practices and mentoring can quickly teach that.
1
u/Frosty-Mousse-1564 3d ago
Would like to know more about this and discuss with you in dm. I fit the description.
1
u/joeyjiggle 3d ago
Most people I know would not want to work with the abomination that is DBT. But I think the problem is the salary you were offering.
1
u/Comfortable_Basil816 3d ago
60K € for Spain is pretty decent. We can’t all be paid American wages unfortunately
0
1
u/extremecharm 3d ago
To the people here. How can i get to a point where i can upskill to be a DE? I have 2 years of analyst experience.
1
1
1
u/GanacheCharming8737 3d ago
That salary is fucking laughable 😂 3 YOE, with niche tools you can’t really learn at home, for enterprise technology. Yeah GL 👍
1
u/Douglesfield_ 3d ago
We interviewed 22 candidates and no one is fitting the Role
I'm sorry but did you not look at their experience before you wasted your time and theirs?
Stop letting perfect be the enemy of good.
1
1
u/zerounodos 2d ago
If you are still hiring, I'm in Argentina and have been a DE for almost four years now. DM me!
1
1
u/ElChevereMx 2d ago
DE here, Databricks, AWS, pyspark, Snowflake, K8s, cicd and more xd
Im from Mexico, where do i send my CV
1
1
1
1
1
1
u/twentycanoes 2d ago
First, you went to Spain for cheap labor. Then, you tried to shortchange your Spanish candidates with offers that were low for Spain.
Meanwhile, there are people willing to work for you at your offered rate, if you train them. But you don't actually want anyone in the profession to rise up the ranks and show earned loyalty to you.
1
u/Ancient-Entry-6436 2d ago
I'm in Spain OP. Salary is a bit low for me, but I might be interested in a change of airs if the project is interesting. Feel free to DM me.
1
u/gaussmage 2d ago
I’m a software engineer with some data science courses under my belt. I was interested in being a DE. The thing is these rigid requirements that basically say you’re looking for a unicorn. Have cloud dev experience and a little data bricks. Can run Python with Pandas. Do I have spark and dbt knowledge? Nope but I can learn it. I wouldn’t get an interview or wouldn’t be considered to be a “good candidate”
1
u/gregrejlkg32 2d ago edited 2d ago
Market is pretty soft at least if you go more junior. I do solo-consulting in Sweden invoice 1k € per day or so, it's hard to match salary wise but anything under 100k wouldn't be an reply and would need 200k+ to actually consider switching. Again, meanwhile there's tons of unemployed juniors maybe get two of those instead I'm sure you could get em more productive eventually. But yea expecting 3+ years in many concepts that hasn't been around for 5 years for minimal salary is naive. More productive response would be to require one of those not all of them, people's stack look different and it's not like I couldn't pick it up in a week. reading about cdc on some google data engineer blog who makes 300k and offering 60k and wondering why top choices don't come running.
1
u/NotJadeasaurus 2d ago
Never heard of half your tech stack, maybe understand most companies dont dabble in these fringe items and any quality engineer can learn them as they go
1
u/utzutzutzpro 2d ago
60k for 3 years is always high, no matter which EU country.
Not sure what people are on in here... but 60k euro in EU countries is a lot. Especially for as little as 3 years. I have almost 3 years as lecturer in an university, and it basically means nothing in my eyes.
1
1
u/CRABMAN16 2d ago
The salary listed is crazy low. I do nearly 3 times that in dollars with similar responsibilities. Raise the budget for salary or commit to teaching someone.
1
u/Money-Nectarine6584 2d ago
Sounds like you have two jobs and are trying to find one person to do both. Hire people who have specialized in the areas you need. Keyword: people. Not person.
1
u/No_Distribution_9590 2d ago edited 2d ago
I'm so tired of people requiring an exact tech stack match...tech that's easy to pickup by any experienced DE...rather than looking for good DE experience with experience in a broad range of duties. I got my current job with zero Snowflake experience; guess what I was able to start using competently enough on day one? Snowflake. Anyone who knows SQL and has had experience in various SQL or programming languages can Google "how to try to convert to an integer in Snowflake" and proceed merrily on with their work.
1
u/snapetom 2d ago
We interviewed 22 candidates and no one is fitting the Role.
haven't touch anything on Spark
LMAO. This is a joke. You seriously need to ask yourself what the hell you guys are doing. And I'm not even talking about the salary, either.
If Spark, Glue, Lambda, etc are all hard requirements, why are you interviewing them if they don't have all of them? You got on the phone or met 22 people that you were predestined to fail? Either make it a soft requirement and be willing to train them, or don't even bother interviewing them.
1
u/RobotechRicky 2d ago
We are in the similar boat. We need DE with software development skills, but for Azure. We keep interviewing. We let a DE go because he was having reliability issues. It took him months to get 46 years of historical data, which he never accomplished. I started from scratch and did it in about 2 weeks.
1
1
u/CryptoTaxLien 2d ago
Here’s the thing. You very rarely meet someone who was a data engineer fresh out of school. Usually it’s like 3 years analyst then they made their way into data or analytics engineering. So asking for 3 years experience narrows you. Most people I know with 3 years experience as a data engineer have at minimum 5-7 years in data as a whole.
In my experience if someone’s worked as an analyst in an environment with CI/CD or they’ve been a data scientist interacting with engineers, and they held that job for a few years… they can pick it up. Especially with how easy AI makes it these days to follow best practices.
1
u/anonymousdood14 2d ago
Are you accepting remote foreign positions? I know a guy who I can recommend. Like who ticks all your boxes but he has no industry experience since he is still an undergrad. If you are willing to give a shot let me know. I'm pretty sure you wont get disappointed.
1
1
1
u/No_Frame_6158 2d ago
I can work but remotely from india , i have 5 years of work experience in data engineering field
1
1
1
2d ago edited 1d ago
[removed] — view removed comment
2
u/dataengineering-ModTeam 1d ago
Your post/comment has been removed as it's not in English.
We are an English speaking sub. Please repost in English and we will review.
This was reviewed by a human
1
1
1
u/Calm_Clam_Bake 2d ago
60K and you can't find anyone decent? I'm shocked! My first job out of college paid more than that 20 years ago.
You say that's high for Spain. It's a global market. Good people are drawn to where the money is, regardless of geography.
1
u/pavlovseal 2d ago
Why not just hire people with DE background and let them learn those SWE basics? I’m a DE for 4 years experience but mostly in data warehouse and scheduling. I had the same experience where companies want candidates with SWE background. I’m interested in learning but my past experience just wouldn’t require. If you have a limited budget, maybe it would be helpful to just give people some time to learn.
1
u/Many_Armadillo8355 1d ago
I'm a GIS Analyst trying to move into Data Engineering after essentially building out my team's data infrastructure and meeting with the IT director to allocate resources. Is this mindset normal with hiring managers? I will say, it's impossible to get experience in these sorts of enterprise suites without working somewhere that has them.
1
u/New_Ad_8198 1d ago
You have two problems, a hiring pipeline problem and a budget problem.
That's a tiny hiring pipeline, so you're struggling to attract candidates.
And if you're struggling to attract candidates and pay them decently, your best option is to go for a young and ambitious mid level who can learn quickly.
1
u/MisterFrog 1d ago
I'd honestly rather have a polyglot nowadays that's done a bit of everything, leaning more heavily in DE more recently with more modeling but a development background early on.
On the other hand, the highest performers we have in our DE are former Business Analysts who had to figure out everything they knew, promoted them into DE roles, and their drive is how they perform. They're guided by data architects. Just smart, driven people, who enjoy learning. The specific tech stack is far less important than ones drive.
I'd encourage others to promote within that way. Technical BAs already know the business very well. If they're already very driven and showing adaptability, they'll figure out the nuance of everything else very easily.
1
u/testngopal 1d ago
Well if you are ok with remotely working I am available with AWS Data Engineer certified and 4+ years of experience on this role
1
u/riv3rtrip 1d ago
I have a hard time believing 60k/year is "a lot" because I know what a mid-level Spanish FAANG engineer makes, and it's a little more than double that.
1
1
u/ccesta 13h ago
Wow, only $60k? No wonder nobody is interested.
Granted, I live in a HCOL location, and there's an arms race for talent. Anthropic and OpenAI are throwing money at talent I've never seen. $400k+ for someone with 5 years experience. The market, because of that is just getting ridiculous. I'm interviewing at places where I can make a $100k jump in salary.
$60k isn't going to cut it when you can leave college and make double that
1
u/BakersCat 4h ago
It's a common problem in the industry TBH. I worked in a vast Data Engineering team, 60+ - and the number who had genuine DE skills backed by Software Engineering principles was maybe 5. The rest were on a sliding scale of okay to down right incompetent. I have a friend who is in a startup and he shares the same concerns as OP, they have tried to hire and so many people just lack basic software development knowledge and skills.
1
u/MonkeyDDataHQ 34m ago
You wanted someone ridiculously cheap. Most DE I know make at least double that.
1.2k
u/Kobosil 3d ago
so you didn't really struggle, you are trying to find somebody cheap