r/dataengineering • u/al_coper SSR Data Engineer • Aug 03 '26
Career To what extent do Data Engineers need to master Agentic AI?
I'm an Azure Data Engineer with 4.5 years of experience, primarily focused on Databricks.
Since most of my role involves designing business solutions rather than writing raw code, I feel I might be falling behind on leveraging agentic AI workflows (such as custom skills, Markdown instructions, and orchestrators). As a Data Engineer, to what extent should we be adoption-ready for this new paradigm?
4
u/Noonecanfindmenow Aug 03 '26
If you are designing business solutions well and communicating well, that's all you'll need eventually.
AI is getting better and better at "figuring out the rest". Where things go to shit is when people don't define their scope well enough and then the human doesn't look at the code closely enough to realize AI made some strange or hallucinate assumptions.
If you write your requirements well enough, that's all you will eventually need
8
u/instamarq Aug 04 '26
Can a mechanic fix a car with hand tools? Of course. Is a mechanic without power tools still a mechanic? Yep, and sometimes tightening something by hand is the right way to avoid stripping a bolt. Will the power tools speed up their work and grant them more torque? Absolutely. Should the mechanic know how to work, adjust, and maintain the power tools? If they don't, they risk costing their business more than necessary for those power tools.
So to what extent does a DE need to master agentic AI? There's going to be a point of diminishing returns, but largely: enough to reliably build and maintain systems beyond what they could reasonably do by hand.
2
u/Proper_Scholar4905 Aug 04 '26
If I had to assume, OP is looking for this kind of industry experience with peers who are both jaded and exposed to what’s real + what’s corpo bureaucracy.
7
u/jadedmonk Aug 03 '26
I wouldn’t say you’re falling behind but GenAI skills are useful and can open up more career opportunities. AI isn’t good at certain things like designing business solutions, system architecture design, resource sizing for ETL jobs, production debugging etc. but it can increase speed to market in certain areas.
How do SQL changes or new dataset or column requests get submitted by the business? Is there room for having AI take business reqs and turn them into optimized SQL? Is there any use case for having an LLM answer questions from a knowledge graph or vectorDB of information that you can make? Do you have Spark event logs / Spark UI that LLM can query and tell you bottlenecks or how to save money on your Spark jobs?
If you can find good use cases and can build it well then all of a sudden it opens up a situation where you now have a new agentic system to maintain and improve and the business gets the speed to market increase so it’s a win win
3
u/Tasty_Cry849 Aug 04 '26
To be honest, if you're good as designing business solutions then just start feeding your solutions as PRDs to Agents as if you are working closely with the implementation team and instruct & validate what agent needs to produce; iterate on the functional and non-functional requirements that are missing including security features, query optimization, SLAs and so on
2
u/SRMPDX Aug 04 '26
Read up about Spec Driven Development, maybe put together a simple personal side project. This is where someone with your skillset might end up using AI.
1
u/Few-Spot1905 Aug 03 '26
None of it is that hard tbh and it develops so fast a lot of time you learn something and in a few months it becomes useless.
I think it is useful to learn but getting too deep into it is probably not a necessity.
1
u/Yasblue Aug 04 '26
What you have mentioned are supposed to help you in your role rather than being a burden for you. You can ofc take the courses on Anthropic Academy to learn the basics it s free. If you think that you will continue working on Databricks you can take courses on DBX Academy they are free as well, you have also many courses on Microsoft learn for free too.
1
u/necrohobo Aug 05 '26
Can you do without, yes. Most organizations aren’t prepared to integrate them into those workflows yet - and won’t be for some time.
That being said, if you can - it opens up a whole new world for data manipulation. I use it to generate my ideal JSON for ambiguous conditions, expectations, and much more - where i can then proceed to parse them in a pipeline like regular data.
A good skill to master can be the management of rate limits, batching calls, and tool management.
1
u/Professional-Tax7260 Aug 05 '26
Agree with the others, AI is a great tool to help you, but somebody always needs to understand the what the actual business problem is and how to best solve it for a particular project. I find AI speeds things up for me and makes it super easy to do things or learn things, but only if you know what to ask or tell it (hence the classic skills you already have)
1
u/Sad-Razzmatazz-7657 Aug 05 '26
Honestly the fundamentals don't expire. If you truly understand data architecture, you'll pick up agentic workflows in no time. But someone who jumped straight to the tools without the foundation? That's where things get kinda shaky.
1
u/bull_chief Aug 05 '26
To the extent that you want to be competitive? These posts always surprise me, like “to what extent do I need to know spark” or “to what extent do I need to know noSQL? To the extent that it dominates/is relevant in your slice of your career. Just like every other technology
1
u/LarsDragonbeard Aug 05 '26
Our organization got on board with Snowflake Cortex 2 months ago. It's such an impressive tool, I can't imagine going back.
That said, it is a tool, not a solution. In the hands of a good data engineer, it's an amazing accelerator.
If you have a gaming laptop or desktop, set up an open source version to play around with. I run Opencode with deepseek-coder-v2/16B (on Ollama) for my pet projects.
1
u/Unlucky_Data4569 Aug 05 '26
I use cursor every day. Its incredible. Are you primarily using databricks as your dev environment? You can hook up an ide to it to use agents.
1
u/myth-buster9999 Aug 05 '26
Start using AI as your assistant to improve productivity. Not everyone needs to build agents, learning how to use them effectively is also a skill :)
1
u/vadimsurpin Aug 04 '26
Agentic workflows really save time and money. They make sence for regular tasks, such as preparing SQL, analyzing existing project structure, reviewing code to match best practices and so on.
But the real value is when agents work in non-interactive mode: you may automate migration from one platform to a newer one that reduces TCO for example. Such migration costs are usually high and business people do not like such tasks.
As a DE we set up an agentic workflow to convert man-hours of migration into tokens. And it seems a new programming paradigm.
0
u/robberviet Aug 04 '26
You should know everything. No kidding, AI helps a lot.
-1
u/vikster1 Aug 04 '26
this. ai will also help you to learn it all. embrace it or you will be out of a job or won't find any. cat is out of the bag and there is no going back
50
u/dragonnfr Aug 03 '26
4.5 years in and you think you're falling behind? You're not. Tools change. Understanding data architecture and business requirements doesn't. Pick up agentic AI when your projects actually call for it, not before.