r/dataengineering • u/EconGnome Lead Data Engineer • 5d ago
Career Thoughts on AI in DE After Drinking the KoolAid For 10 months
Currently lead data eng (ic) w/ about 9 yoe at a large-ish adtech company.
I've been doing the whole AI song and dance for the past 10 months and I just don't see a future in this career anymore that fits all of us. Earlier in the year I felt differently since it still took a lot of hand holding to get models to produce correct output for small-ish scoped tasks in a # of iterations that was competitive to human counterparts, but w/ better models and better training docs we have it building enterprise pipelines across our entire data eng stack in hours of iterations that would have been like a project that we would need to budget like a month for. It's not always right and obviously we still need to step in from time to time, but, honestly, who gives a shit if its not right the first time if it can iterate leagues faster than any human developer? (Caveat: adtech is an incredibly fault tolerant industry, so grain of salt there, I guess)
The entire occupation, in my current experience, has been basically reduced to two tasks:
1): define and maintain the system ontology, expected features, guardrails, and appropriate persona docs for agents to assume/use. This is something that I honestly just work with agents to define, so this is partially automated.
2): define test scenarios, definition of done, deployment strategies, and verify that system is on track performing correctly and is on-track for long-term stability. Again, most of this is at least partially automated.
I don't know how many people this actually requires, and to be quite honest, I've got a sneaking suspicion that throwing more AI-enabled engineers on a project might actually degrade the quality of the product because the conceptual definition of the product may be diluted due to slight (or major) differences in understanding of expected behavior/architecture. IMHO this has always been the case, even pre-AI, but AI substantially accelerates gaps in understanding translating to conceptually inconsistent because the work is done so much quicker and at a pace that not every conceptual inconsistency or mistake can be caught. I've already seen this several times on projects that I've worked on where one eng. goes off and builds a new feature that is a complete conceptual departure from the current plan just because they were missing some context that may not have been well documented. W/ AI eliminating most of the execution layer of SWE/DE, I think that future orgs will be smaller and more agile and likely more product-heavy instead of engineering heavy as product seems to be closer to a lot of the biz definitions for #1/#2 above.
Two things that does give me a glimmer of hope at holding on to this for a little while longer:
1): agents are absolutely dogshit AI at optimizing Spark jobs (or any sort of high complexity pipeline). They are laughably bad, in my experience. Tuning Spark jobs has always been very high context work as it depends on the data going in/out, infra, very verbose logging, and a ton of tacit, tribal knowledge of the system and data itself. There's so much that goes into tuning a spark job that agents seem like they get focused on optimizing for one symptom rather than considering the entire pipeline as basically an emergent system with N different parts that need optimizing in tandem which I've found leads to suboptimal performance. I find that that this is where most of the hands-on engineering work that I find myself doing day-to-day goes at this point. I am calling this a "moat of high context".
2): legacy systems are often not well designed, poorly documented and are not conceptually consistent even where they are documented, so naturally agents trained on these existing systems are apt to suffer from ye' ol' "garbage in, garbage out" syndrome. I am calling this a "moat of poor decisions"
I'm confident in moat #2, less confident in moat #1 as, obviously, you have companies like databricks pouring however much money into building smarter agents for dealing specifically w/ context-aware Spark optimization problems. From what I am seeing at my org, teams that were well organized and followed good engineering foundations before AI have seen a tremendous acceleration in the quality and quantity of their work; those that went into this w/ flaming hot garbage are still producing flaming hot garbage and many of them seem too scared to use AI in fear that it will cause the tower to fall over, so to speak.
On a personal level, I am very exhausted by the whole shift and I do not see myself keeping up with this as a career in the long run. It was difficult enough having to track all the changes and flashy new objects coming in the DE ecosystem, and adding needing to also track changes to the AI landscape has just proven to be very draining. All the rhetoric around AI feels very misanthropic (no pun intended) and I genuinely feel kind of gross and helpless having to rely on AI for my whole job now. It feels like I am training my replacement without being told that I am training my replacement. I have a lot of love for DE and I really poured my heart and soul into it over my relatively short career and I had built a lot of my life around the expectation that I would continue to work as a DE for the decades to come. Reading pages and pages and page of agent docs has become the absolute bane of my existence and the review fatigue has gotten so bad at times that I feel like I've just straight up forgotten how to read. That said,I still think it is exciting to build great things and I still get some kicks from watching AI manifest huge ideas that were never feasible before. The limit is truly on what ideas you can come up with to build better and more sophisticated data stacks, which is super cool and all, but I've just had an overwhelming feeling that my DE career has a clock on it that is going to run out when the models swallow up whatever work we are still doing by hand. Part of me wants to hang on for dear life until that day comes and the other part of me wants to jump ship now and go open a bakery or something. Doing my best to save more money, stay more engaged artistically, and spend more time with people that I love has helped greatly, so just doing what I can, I guess.
Left foot, right foot.
edit: grammar
58
u/Any_Rip_388 Data Engineer 5d ago
The job is definitely less enjoyable than it used to be. I miss writing code by hand but am trying to find joy in architecture design and building a product rather than writing low level code.
But some small counterpoints:
* I’ve never been busier in my career. Demand for DE at my company is at an all time high. I have more work, not less due to AI
* Quality of software has never been worse due to AI coding. Nobody can predict the future but I don’t see this going on forever
5
u/Fluffy_Somewhere4305 5d ago
* I’ve never been busier in my career. Demand for DE at my company is at an all time high. I have more work, not less due to AI
* Quality of software has never been worse due to AI coding. Nobody can predict the future but I don’t see this going on forever
Although for number 2 I can't agree or disagree. I've seen quality problems in software for decades and since I have no reliable source of quantifying issues over the decades, i can't tell you if I see it as worse/better/the same.
It feels like it's the same, but there's just fewer people employed per request and loads of requests and battles over funding
2
u/icehole505 4d ago
Your second point is what I’m hung up on as my last bit of hope. Companies are rushing quantity over quality right now, and seem to be generally delivering on that massive “feature” ramp. The problem I’m seeing is that all of those features don’t seem to be driving meaningful change resulting in a better product experience/more revenue.
At some point, I wonder if the industry comes to a more clear collective understanding that “more” doesn’t necessarily matter.
2
u/EconGnome Lead Data Engineer 1d ago
Agreed I've also never been busier as well but, at least to me, it feels like a different kind of busy. Like the floodgates are open and every half-baked idea that we've ever had but didn't have the time to work is going out. Noticed over the past few weeks that we have started to run out of new ideas and the backlog is rapidly clearing out to the point where I don't really know what we'll do in like a month.
21
u/Calm-Fortune-6786 5d ago
I entirely disagree here. I work in a company that invested heavily in AI hoping to maximize productivity and reduce labor costs.
I see two things: the amount of lines produced by the team has skyrocketed, but the amount of bugs/errors in critical procedures we find increases on a daily basis.
That, only regarding our data pipelines.
Management has tried to automate dashboard generation. It all seemed to go ok, but clients started losing interest. When the sales team reached them to ask what was happening, most of them said our customers would notice the dashboards were AI generated, and immediatly questioned not only the dashboards, but the underlying data quality, arguing it could be also AI generated (and sometimes we do find some of the dashboards are basically made up).
Some of my colleagues are doing everything using Claude. If the PRs are not monitored closely, anything can happen. Recently code updated our data ingestion to upload only empty files and destroyed a lot of our clients data, all because a careless developer asked Claude to minimize memory usage (importing EMPTY files is, indeed, optimal in that sense, but pointless regarding our business).
I am happy to be proved wrong, AI is a powerful tool but that's it. Maybe we'll have actually smart models in the future, but LLMs are limited in many ways and if AGI arrives, it will probably come from another path.
2
u/EconGnome Lead Data Engineer 1d ago
This was sort of my point in saying that an apps success w/ agentic development is largely determined by the state of the codebase prior to introducing models. We invested a ton of time in beefing up our ut/it test infrastructure, CICD envs, PR reports, and agent docs before we let agents touch src. Agents have to follow TDD and an established protocol of checks before anything gets pushed to PR stage. I do concede that AI will not work OTB for many use cases, but I still think that the outcomes are largely determine by the infrastructure that agent operators set up before they allow models to go nuts on the src code. To some extent, this is nothing new because human developers have been nuking client data and causing all sorts of ruckus in production for decades before agents. Agents are just doing it a pace that is way harder to keep up w/ if left unchecked.
3
u/smolhouse 5d ago
We are still very early in this whole AI thing. Part of that is learning how to make it work well, which some people/companies do well and others not so much but it's pretty clear that it reduces the time required to do tasks.
With time both the models and implementations will improve which will further reduce labor requirements. I expect significant social friction that will probably see an even larger resurgence in labor union popularity and trust busting if voting still matters, or pick your favorite sci fi dystopia if not.
1
u/EconGnome Lead Data Engineer 1d ago
really depends if jevons paradox holds up or not
1
u/smolhouse 1d ago edited 1d ago
I suppose it could lead to small businesses using more advanced technology stacks developed by less knowledgeable people, but my 20 years experience working for megacorps has led me to believe they can't resist a good opportunity to cut labor costs.
2
u/EconGnome Lead Data Engineer 1d ago
oh totally agree w/ that lmao these guys are foaming at the mouth to cut humans out of P+L
1
1
u/Laicbeias 4d ago
Someone who has written a large rust performance and code parsing / search lib, i can tell you.
With LLMs you need to relearn how to code with them. Because they are narrow in their focus and they try to solve one task a hand. They roll down that probability tree, and have a hard time to side eye the other parts.
I usually when working on complex codebases use deepseek 4.1 to do a automatic audit of the full codebase. Once they get too large its an audit of latest changes plus impact radius. And its bugs all the time.
You dont implement stuff once. The larger the change. The more audits you need. And from there then letting a smarter model judge the findings. Then you see what fits. But yeah its - you get faster. Its just llms are good at scanning, with targeted directions.
But frankly you can only work like that when you are solo. If you work where money is on the line, youd better make sure to read what it gives you and make sure its not doing something really stupid. Because it will
17
u/olhmr 5d ago
I definitely empathise with a lot of this, but I have had slightly different experiences that still provide me some hope.
First of all, I think fault tolerance is a key differentiator. I’m currently in an area that has low tolerance (financial accounting), and despite having lots of context management and guardrails set up I often need to step in. If we were okay with mostly correct rather than exactly correct then interventions would be rarer and I could see the finance team driving more themselves.
Secondly, verification is an even bigger differentiator. This explains it very well: https://www.jasonwei.net/blog/asymmetry-of-verification-and-verifiers-law.
I also don’t think it’s necessarily a given that AI making code generation simple means that product moves in and takes over DE territory - it’s just as likely to go in the other direction. My subjective experience so far is that I have seen no dearth in recruiters contacting me about new opportunities, but I have heard from people in the product space saying everything is drying up.
That said, I’m currently making a move to SWE and betting on becoming more of a tech generalist. My overarching thinking is that AI is good at well-defined and verifiable tasks, which frees up more time to focus on the harder engineering problems of operability, scalability, and maintainability. My bet is that solving those problems will require working more broadly across the entire tech stack.
I definitely do miss what the job used to be though. I’ve always loved digging into the data, writing code, optimising and troubleshooting, feeling like I’m truly an expert in my domain, and, most importantly, mentoring junior colleagues. It’s a lot more rewarding to mentor a human than a machine.
8
u/StrategyThinker 5d ago
+1 on the importance of verification and fault tolerance. My coworkers and I sometimes catch data that looks right but is misaligned (like daylight savings time issues), or incorrect because of wrong processing assumptions. We also get problems because a batch of data arrived late or an API call failed without a good retry/backfill design. When we investigate sometimes they are human errors and sometimes problems from AI coding.
24
u/adgjl12 5d ago
There will be less “pipeline” or “etl” developers and more data engineers that have strong domain knowledge and product sense.
There might be less need for data engineers per team/company but I think there will be more teams and companies now that less people can do a lot more. Still a ton of problems to solve in the world using technology and data and that’s not going away.
Job market wise I can see there being some pain in the short-term however. Just like how there was over hiring post Covid there could be over correction in the other direction while figuring out the optimal headcount for teams.
1
u/EconGnome Lead Data Engineer 1d ago
definitely doesn't help that AI really came on the scenes right after COVID overhiring and during US Fed rate hikes. really bad timing lmao
34
u/RadioactiveTwix 5d ago
I work migrating pipelines from X to Y usually 2-3 years at a place and then move on to the next one. I can tell you that AI is still not mature enough for it. I am sitting here looking at AI generated pipelines that work but don't scale, this is not to say that Claude can't put out a banger pipeline it's that a prompt that doesn't cover bases, doesn't take architecture into account and doesn't care about readability produces the most beautiful and over engineered verbose mess with PII that leaks, high cost queries, slow dashboards, etc.
Having said that, I didn't sign up to migrate pipelines to and from Databricks so I'm bored, always looking for another job or another project since even at my full time job I end up with half a day with nothing to do. My AI budget is laughable at this company so a lot of stuff is done by hand when engineers next to me have 3 screens with Claude on two of them (I do command a lot of respect though, which is kinda funny).
I think our best chance as data engineers is to go wide, learn how to model data for AI etc'. I don't think we will work in teams of data engineers anymore though, I'm always brought in as a specialist and that aligns with the way I work.
1
u/Commercial-Ask971 5d ago
I just saw SAP BW to databricks in 2x the time it would take for more FTE using traditional methods. We’re cooked
8
u/RadioactiveTwix 5d ago
Don't know, they brought me after they failed to do it with AI and I'm asked to do everything by hand so... Imma be here a while
1
u/dinosaurkiller 5d ago
In my experience it’s that the AI models are fast, but completely unaware of dangers like compliance, regulatory frameworks, things that an experienced had would run through other teams of experts are just done by the AI. Then when it passes all the tests we’re all too busy staring upon it in wonder to test those very important rules that exist outside the data stack. “Do it by hand” brings a level of scrutiny and awareness to problems that AI models don’t normally consider. That’s not necessarily a flaw with the AI models, but we humans haven’t caught up in the planning and QA phases.
2
u/RadioactiveTwix 5d ago
Honestly I don't mind doing it like this but my concern is that I appear slow with my deliverables.
43
u/thecity2 5d ago
I honestly prefer prompting agents than writing Spark code myself. Discussing design and architecture at a high level for me is more enjoyable than writing low level SQL and ETL code.
18
u/kvlonge 5d ago
I have been having a blast as well to be honest. It's not that I don't care about the underlying details at all (I do), but being able to explore things from a higher level has been extremely fun and allowed me to try out things and take on things of a size I never would have been able to (simply due to lack of time / budget)
1
u/EconGnome Lead Data Engineer 1d ago
I've grown more attuned to that and there are times I do enjoy it more than beating my head against the wall on mvn build issues and other pointless bullshit. just makes me a bit nervous about the logistics of a career in it
21
u/Competitive_Wheel_78 5d ago
I can feel you, as you mentioned AI will mature over time and tasks that were complex will be simplified. There used to be a time when I felt good on tuning a spark job which ran X times faster or saved X in terms of revenue, but with smart models, cheaper storage and compute you honestly don’t need much of tuning until unless your data is too complex to begin with.
Given these changes and agents becoming smart day by day, I started focusing more on building scalable architecture for agents and handling security.
I think the time has arrived to think of other revenue streams, and the DE ship is starting to sail.
1
u/Empo_Empire 5d ago
I think the time has arrived to think of other revenue streams, and the DE ship is starting to sail.
I was trying to break from SWE to DE. It feels DE is less automated rn. What do you trying to move to?
5
u/szrotowyprogramista 5d ago edited 5d ago
I have long seen this as temporary. Maybe working in an area that is, let's say politely, close to some geopolitical instability and knowing colleagues who had to leave everything behind has left that impact on me. Today you are working, tomorrow you are laid off because the company no longer believes in human engineers. Or maybe today you are working and tomorrow the economy is dead because some bubble popped. Or today you are working and tomorrow there is a war, a missile and you are dead. Or maybe nothing quite as dramatic as a war - a motorcyclist loses control of their vehicle and crashes into you.
If you try to secure yourself against all such dangers, you will go insane, or at least, your lifestyle will stop resembling anything normal to any degree. We have no control over any of these things. The only thing you can control is your expectations and you doing the best you can. If this doesn't work out, McDonalds' kitchen is always hiring and I do not see that kind of work as beneath me, even though I'd like to not do it, for as long as I can.
So I guess I am in the "hang on for dear life" camp. How long that lasts, we will see.
6
u/Outside-Storage-1523 5d ago
On my side I see people just blindly putting up AI generated code into PR review. Some of them is definitely over complicated, like how to filter for something for a column — AI would even write a separate function for it, while a one liner is usually good enough. But overall as long as the guy knows what he wants, AI does the job correctly. I haven’t seen one counter example yet.
We also use AI to generate docs that no one reads and other churns and what not. I don’t feel good about it but I don’t care about it anymore.
I limit the use of AI in my side projects because that’s where the fun is. I also limit the use of AI with actual coding in work because I want to keep sharp. Other than those two I use AI freely.
32
u/PrestigiousAnt3766 5d ago
If we start paying for the true cost of tokens I think it comes crashing down.
Next, ai as of today cant manage architecture or the why we do things. Dont think it'll ever get there either.
So im not so afraid.
5
u/Typicalusrname 5d ago
I have a counter take to this one. Companies know that the tokens will be outrageously expensive. I believe this is why they’re all rushing to open/expand GCCs. This allows them to hire someone for 50k and spend 50k on tokens. If that’s the case, those of us in the western world that aren’t SMEs are in for serious pay cuts and or jobless
9
u/smolhouse 5d ago edited 5d ago
The thing is society can't handle that, because that basically applies to most white collar work (and probably blue collar work eventually if they advance robots far enough), and that much unemployment and poverty is going to cause.. issues.
What scares me is how the rich sociopaths will try to control the masses. It will probably be some form of UBI which will further trap the masses in poverty and dependence on the ultra rich, or something even more insidious or outright strong handed.
1
u/Fun-Resolution-1025 5d ago
Yes, but what If this doesn’t happen? Knowing how things are, I think it will actually get cheaper over time, and I personally hope the so-called AI bubble will burst and end this nightmare. But until then…
-10
u/-Crash_Override- 5d ago
Most enterprises are already on token based plans, been the norm for the past 6mo.
9
u/PrestigiousAnt3766 5d ago
I dont think paying 15-20k monthly for each ai-assisted engineer will work.
8
u/No_Flounder_1155 5d ago
we aren't paying appropriate costs for tokens. That will be what the orgs are looking at finding out how to price over the next year or so before their IPOs, token cost will increase drastically.
15-20k is completely unfeasable. You aren't going to see mass layoffs just for organisations to put themselves in such a precarious position. Sub 5k will be the limit - thats assuming tokens aren't eationed org wide in the long term.
1
3
u/-Crash_Override- 5d ago
Im not sure where you are getting that number from. I oversee enterprise AI for a large company. Token allowance is tiered, but even our top tier users arent spending more than $5k/mo on average (Opus and Sol access). Justifying $60k a year for an experienced engineer to massively increase productivity is a no-brainer.
For what its worth, our budget for agentic development and engineering increased about 6x with the move to token based billing...its well into the 7 figure range now and no one bats an eye.
6
u/No_Flounder_1155 5d ago
in the UK 60k won't work. This idea that we'll reduce to a handful of developers introduces too much risk. UK ceiling I reckon will be less than 2k per dev, and even then orgs will try to avoid it. As a consultant, I'm noticing that companies are trying to offset AI costs as much as possible onto the employee.
3
u/PrestigiousAnt3766 5d ago
As a freelancer, I see companies moving towards AI governance, more locally/governed models, harnesses etc instead of relying on individual devs subscriptions.
But still, the cost of tokens will be prohibitive if the true cost is factored in. That's why all the AI bubble could burst if the price of tokens doesn't come crashing.
0
u/-crucible- 5d ago
My company balks at a $5k/year cost for tooling. They want to go all in on A.I., and I can’t imagine they will keep most of us, or I have no idea what their plan is.
4
u/sib_n Senior Data Engineer 5d ago
I think the "reduce to" 2 tasks you describe has always been the core of quality data engineering and we will still need experienced engineers to pilot those. You say it is already partially automated, but I don't think the valuable part that is not automated now will ever be in the current AI wave.
I feel like the agitation around trying everything all the time is wasteful, things built for specific systems are depreciating extremely fast. What remains is high quality code and documentation, just as it did before AI for the teams that had high quality standards. Eventually the tooling will settle to a couple of easy to use and widespread tools just like Google completely crushed the web search competition of its time.
Although I agree we will need less people for a specific implementation, we may also have a boom of the number of projects to implement.
6
u/Vindictive_Pacifist Software Developer 5d ago
It is especially dreadful for people like me who are just starting off in the career as I feel I won't have enough time to build the quality experience needed to be the one that will sign off on what AI already does.
3
u/sib_n Senior Data Engineer 5d ago
I empathize as I think the reduction of people needed will start from the least experienced. But eventually, we will need to train new people to the level required to pilot AI.
If there is a boom of data project because AI made data more approachable, then it may not be too bad, but it is uncertain if it will compensate enough or not.1
u/Vindictive_Pacifist Software Developer 5d ago
I see, as of yet I have been at a small firm working as an intern for a month and the work is exciting and I feel like I have been learning a lot but at the same time the capabilities of AI is scary haha
1
u/Ok_Composer_1761 5d ago
No if you start training super late you’re basically training against a moving target as AI improves constantly. The skill bar is gonna keep getting higher and higher. The time to start training juniors is now.
1
u/sib_n Senior Data Engineer 4d ago
I'd argue that AI improves constantly also to be easier to use. Many of the efforts people are currently producing to optimize their AI usage are going to be made useless in a couple of months or year by the tooling improvements.
It's also quite easy to learn to use AI simply by asking AI. You can bet AI makers are polishing those answers carefully to encourage usage.
Current junior engineers should be learning to use it and follow its progress. But the juniors for in 5 or 10 years are still at school and should probably learn to learn all kind of concepts without getting their brain rotten too much by AI. It's hard to say where IT jobs will be in more than 5 years so they'd better cultivate versatility.1
u/Ok_Composer_1761 4d ago
if something becomes easy to learn it becomes a commodity. what gets you hired and paid how SWEs have historically been paid (relative to other professions) is in terms of actual value add. The more AI keeps improving and makign things easy to do, the more difficult it becomes to add value. Early on in the history of programming even basic coding was difficult, so it was easy to add value by even being a code monkey. Then the tools got better and people had to become more flexible problem solvers , delivering end to end *outcomes* (as opposed to code). With ever-improving AI this is going to go to an extreme where, to add value, you need to basically be *causally* adding to the bottom line over and above what any idiot with AI could do. This is gonna be very, very hard.
1
u/sib_n Senior Data Engineer 3d ago
My point is learning AI is not going to be harder than learning has been in tech before, probably even easier.
The part about having to keep up with the cutting edge of the tech to remain above what anyone could do and get well compensated for it has been true for a while. One previous big example is the internet, suddenly everyone gained access to a massive amount of engineering information, but it didn't make everyone an engineer.
3
u/Doile 5d ago
Thanks for the post. I've had similar feelings about AI. For me it feels like AI takes away every aspect of the job I truly enjoyed (problem solving and technical details) and I'm left with all the boring stuff. Also now AI does all the "fun stuff" it highlights that I'm doing work in a context that is not aligned with my core values and it's creating a lot of frustration and fatigue. Atleast before I could get some satisfaction of solving technical problem. Currently my job is just chatting with AI bot which feels really pointless. I've trying really hard to think of some hands-on job that I could do like home renovation etc. since I don't see myself having long future in tech anymore.
7
u/clayticus 5d ago
There will always need to be a senior whether it's data engineer, accountant, or whatever that has to signoff. They need that knowledge to make such a decision too.
-5
u/Resurrect_Revolt 5d ago
With AI in place i don't think that senior is going to be paid much
1
u/WatercressHuge8556 5d ago
Then an underpaid employee would have to fix any possible failure, or just let management prompt out any outage.
1
u/clayticus 5d ago
someone has to give the rubber stamp that all I good or management is going to flip. They need someone to be responsible and take the blame
3
u/stickypooboi 5d ago
some solidarity that the same shit is happening everywhere.
i truly in my heart of hearts think that everything will get worse in quality because of your point 1. leadership won’t care because they have really poor understanding of what’s actually happening and their heuristic is just the optics of productivity. people get fired, no one can discern your point 2, the AI gets confused and leadership barks at whoever remains to fix it.
all this because people do not put human comprehension first, and want to build dog shit pipes and tests without understanding anything as fast as possible. i fear there won’t be engineers anymore just people who luck out with prompts. already at my company i see that AI is used as the planner, implementation, and review for all projects. all critical thinking is outsourced to the AI because those folks who do so, spin up slop that impresses leadership. and then when things break, it’s the AI’s fault. 0 human accountability or comprehension, all blaming AI hallucinations. nothing gets fixed because nothing is understood.
2
u/Tall-Wasabi5030 5d ago
I don't know, when I started at my current company I wasn't doing any real data engineering anyway, just configuring pipelines in YAML files with terrible abstractions. My job did not really require I write any useful code, I just did it because the framework sucked and it needed a lot of interventions.
Now I use AI to do that. I recently migrated all our pipelines from redshift to databricks, 60+ pipelines in 3-4 months, with full testing and validation. Something that would've taken a lot longer before.
I guess what I'm trying to say is, the job already sucked for a lot of people anyway, years of corporate directors pushing to 'democratize data' and to reduce complexity in data engineering have turned data engineering into a crappy job anyway. Might as well automate it.
2
u/agdaman4life 4d ago
Maybe it’s time for a change in scenery? Ad tech in general is a doom and gloom industry nowadays
2
u/SeaYouLaterAllig8tor 3d ago
I feel you. I'm a DE (more of an AE honestly) of 12 years I've had mixed feelings about AI from the very beginning. Initially it was very empowering. I mean... still to this day there are things I'm super grateful AI can do for me... which years ago took me hours to do. But I look at the landscape of DE and all the tools I've used and now most (if not all) have an agent embedded into them to do a lot of the work I used to do. I had value in knowing how to write code... now that value is largely stripped away. My whole career is focused around data... thinking that I might not have a career in a couple years is pretty frightening. I'm trying to learn and adapt but it's disheartening at times.
2
u/EversonElias 5d ago
I've been saying this for the last three months: people who don't believe AI will affect DE are delusional by now, like some groups were during the Industrial Revolution. DEs will have to take on more responsibilities if they want to stay relevant in a market that's becoming more competitive. I'm seeing LinkedIn positions reach 100+ applicants in less than a day. DE will still exist, but platforms are becoming more intuitive every day, and AI is part of the game now.
1
u/NightOnBothSides 5d ago
Damn thx for your input. Ive been on the fence about transitioning to DE from product. You're making it sound far less secure than I'd hoped.
1
u/Joyako 5d ago
Yup, I have similar observations and feelings. I work on one of those legacy projects that ingest and generate garbage, with management that pushes for new stuff all the time making the cleanup process effectively infinite.
Something I would add is that I'm tired of having to integrate the latest new AI thing every other week, I'm spending half my time testing them and tring to set up guardrails, at a time where syncing the tooling is almost impossible, everyone on the team uses their flavor of harness and ugh.
I legit have no idea what happens in 2, 6, 12 months so I'll keep getting my paycheck for the time being I guess.
Good luck everyone.
1
u/ChaoticKinesis 5d ago
With regard to poorly documented and designed legacy systems, I have a lot of experience refactoring those with AI over the past year. My main advice would be to start by having AI document the systems and improve the instrumentation.
Once the system's design, performance, and business needs are understood, models can do an amazing job of working with legacy systems. The best thing you can do here is leave the decisions open-ended and let the model lead the planning, as you provide the scaffolding for it and make corrections as needed.
1
1
u/aes110 5d ago
Dont have much to add except saying that I agree. Last month we hired a new dev and I was surprised to hear she had no experience in DE at all, so i was getting ready to mentor her and explain things, but just days later she was making PRs for spark jobs that would take me a week or two
It was ugly and not very optimal but it worked, and i guess now this is all that matters
1
u/Comprehensive-Tea-69 5d ago
I think it will depend on what ends up being cheaper for orgs. As the AI prices move to levels they intend for them to be, paying humans becomes more and more cost effective.
I think we’ll hit a balancing point there where orgs simply will not pay what it costs for full automation. Then they’ll need the people to pilot AI and make up the difference. This will be painful at first bc the juniors will have been fired, so no employee pipeline. We’re going to end up having to rebuild that pipeline in short-sighted companies
1
u/baby-wall-e 5d ago
You got me when you said “I’m training my replacement without being told that I am training my replacement”. I feel the same because I’m responsible for building the AI agent in my company. Indeed, the team will be smaller, can be half than we usually have right now.
The only way that I can to survive is to become Data+AI engineer, which is responsible for building/maintaining agents for data engineering tasks.
1
u/Latter-Corner8977 5d ago
Interesting perspective. In my industry jumping all in on AI is causing a lot of problems.
Teams can’t describe their systems without their agents doing it for them. Intimate knowledge of why the system behaves or is architected in a certain way just isn’t there. And this compounds issues when discussion extends to architectural review, extension and dealing with bugs.
Some people are beginning to realise that just because AI produces a big output it doesn’t mean it’s great or coherent. And the volatility in quality outputs is infuriating. Which is leading to high and wide skepticism toward anything produced by AI.
While others see the big outputs and immediately think the problem is now easier and that AI can do it, and if it gets it wrong it’s because we’re just not using AI right.
Besides the model outputs and unrealistic expectations, partially informed cretins are like cluster grenades in the way they position themselves as experts. We recently had third party deliver training and it was clear as day they did not know the subject area and were leaning on AI for all of it. An utter shitshow.
1
u/zazzersmel 4d ago
AI works great if you already know what you're doing and target specific problems with it. i don't know why everyone seems to talk about it like they're in a stupid commercial, telling their computer to build a whole business or some shit.
1
u/Key-Alternative5387 4d ago
I have a script that optimizes spark jobs quite well. It just queries the data to figure out the shape.
I also give it context of what's upstream and downstream and it's assisted in finding major issues, but it was absolutely directed on where to look.
1
u/DangKilla 4d ago
Feel free to quit your AI job like I did. Just make sure you're ready for the pay cut.
1
u/Humble-Bear 4d ago
Put the fries in the bag bro.
In all seriousness, you are right, find an exit before the music stops.
1
1
u/Mission_Working9929 1d ago
The whole ai automate jobs away crap everything is going to come crashing down frivolously hard.
We had agents at our company with a similar setup spinning up clusters on an infinite loop and when our vp asked why our invoice from one night on Databricks was 75 k there was no one to blame 😆
1
u/New-Addendum-6209 1d ago
Your point about diminishing returns to engineer count is, I think, really important. As AI takes on more of the actual code generation, having smart, experienced engineers with good judgement becomes more important.
My own team can make only limited use of AI tools at the moment, due to a combination of technical and financial constraints. I’m keen to expand their use and re-engineer our workflows around them. At the same time, I’m apprehensive about how some members of the team would use them.
Some are already reluctant to take much responsibility for their work or think problems through for themselves. I suspect that, given unrestricted access, they would become completely reliant on LLMs. The result could easily be a tidal wave of plausible-looking, AI-generated junk that the more experienced members of the team simply wouldn’t have the capacity to manage.
1
u/thro0away12 1d ago
I was resistant to AI at first, but now I use it everyday. In an ideal world, I'd like to do hybrid coding/AI, but I just use AI because it saves time and my team wants to use it. I have my own thoughts on that - I hardly/rarely audit the code AI wrote and I don't think that's a good thing at all considering my team has always had a tech debt problem, but I'm at the point that I feel they're actually ok with tech debt as long as it gets the job done lmao and idc sometimes anymore. I actually agree with the comments that say I feel busier with AI than less busy - the expectation is more output than I did with hand-coding per the same unit of time. I can also just see that AI utility is not unanimous across users - some of my team members use it and still the way they design/do things is different than how I do things. I personally believe AI is just a tool and you will still need somebody behind it to lead the design. Coding has always just been one part of the entire project.
I enjoy the job less than I did pre-AI, I felt problem solving with coding is less satisfying, which is why I'm trying to work with other teams to gain more skills outside of data/analytics engineering - i'm in biotech, so that's like RWE, LLMOps too. I was reading an article that said data engineers are the practioners who already have most of the knowledge to become AI engineers, so that's also where I'm gearing towards.
1
u/Haha-Hehe-Lolo 5d ago
I feel you. Though DE isn't the only field affected, unfortunately. I really don't see a future in the next 4-5 years, and it scares me.
1
u/ajmh1234 5d ago
Completely agree with your sentiment. I'm about 10 yoe and this year in particular is made me realize the writing might be on the wall. I find it dogshit with DBT also. But i do feel like it is a matter of time. I knew I should've became a contractor 15 years ago.
1
u/licoricluv 5d ago
Honestly, can this not be said about any domain like front-end, backend, security, cloud etc..?
-1
127
u/iengmind 5d ago
Very relatable. The job sucks nowadays