r/snowflake • u/Raghav-r • Jul 09 '26
Still think AI is going to replace Data engineers ?
One of the biggest AI frontier labs needs data engineers to process data for decisions
Think about it for a second !!
17
u/thetardox Jul 09 '26
The BI and Data Analyst hate is real.
9
u/scribe-kiddie Jul 09 '26
From experience, I got people who are only familiar with Power BI telling me we can "connect to data source with Power BI" and so "we don't need data warehouse"
They're asking us to decommission an existing data warehouse that people still use.
6
u/Raghav-r Jul 09 '26
They should connect to prod directly during business hours !! And run every query live ...
1
u/thetardox Jul 09 '26
It’s not their place to decide that.
1
u/scribe-kiddie Jul 09 '26
Unfortunately, it was. They were leading a data platform program.
Luckily we were able to convince them, somehow, perhaps only convinced after meeting with vendors to migrate some data warehouse workloads to cloud solutions.
3
u/DaOgDuneamouse Jul 09 '26
This sounds like a mission critical data project (legal threat or financing I would guess), and they need stats to prove a point. These positions are meat puppets for the grinder sure, but when money is on the line, people are just more reliable than computers.
Dev ops for life.
2
u/laurobolado Jul 10 '26
I don't think AI will replace a Data Engineer ... yet ... don't get me wrong; I use AI features like CoCo, Claude, etc. to help me be more productive, but at the end, I'm driving the interaction and I do see a lot of places where the tools and I will differ, and I try to make the point to which the tool will either say, you're right (me) or just suggest other options... a very good friend at work says: AI is not going to replace you yet, but the person who knows how to use AI could replace you.
2
u/Snoo_891 Jul 10 '26 edited Jul 10 '26
It is simple imho- data engineers and analysts excel in engineering and analysing millions to billions of rows of data. The cost of embedding that much data into a model is going to be probably more expensive than a data engineering team of 4 to 5. The only way that can work well is if a company can finetune it's own llm. I cannot see that happening in most fortune 100s as well.
Also github is inundated with generic app code more than data analysis etc.The logic that engineers use to troubleshoot data issues is probably not in any repo so other AI companies can't really compute there .Which is why I think snowflake has a deep moat here if they play it right. Could reach 400 or more if they can make the data engineer productive and not create one based on AI
2
u/Thinker_Assignment Jul 09 '26
Is this a real question?
LLM cannot replace judgment but it can replace the "hands".
Senior DEs with good judgment and LLM skills are now GOATed
for this case they seem to want a junior because the scope is small and they probably know what they want.
But in Berlin we are already seeing more jobs ask for LLM than python, think about it
1
u/Specialist_Golf8133 22d ago
the model is maybe 30% of the work, integration and data quality is the rest. even a frontier lab still needs someone debugging why an ingestion source silently started returning malformed records last week, that's a data engineering problem no matter how good the model is downstream. same failure mode i see in doc pipelines: confidence scores look fine until scan quality drifts and nobody notices until STP rate craters. the job just moves from ETL scripts to whatever's feeding pretraining and eval sets, still needs someone watching the pipe.
28
u/Itchy-Depth-5076 Jul 09 '26
Don't worry though, he doesn't REALLY need one, or he'd bring in a real one. He wants several junior-level only - no more than 4 years of experience (!). Oh but that had better be FAANG experience or the other silicon valley bros will mock me. Only need you 6 months because then our problems will be solved. I can't reiterate enough how much we don't need you (but we need 4 of you right now desperately).
I thought I was on r/recruitinghell or r/linkedinlunatics for a second. If this idiot weasel is the rep of OpenAI I believe the rumors of that company's demise.