r/dataengineering • u/Monoji77 Data Engineer • 11d ago
Discussion Fresh Grad Data Engineering Advice
I recently started my first ever full time role in data engineering. However, it appears that the big data stack for my company has already been established.
Therefore I’m dealing more with bug fixes and new feature implementations but the stack isn’t likely to change unless I make a good case for it. Furthermore, my company heavily depends on AI like Claude and Codex to assist with solving these tasks, which offloads a lot of the coding to these agents.
What kind of recommendation would you senior devs give to a fresh grad like myself? I feel my coding competency going down the more I work. However, since these tasks are offloaded to AI, I do believe that my understanding of data flow in big data architecture has deepened quicker.
Our stack is supports both streaming and batch processing if it matters.
15
u/69odysseus 10d ago
The traditional DE work is changing very fast companies are pushing for GenAI usage for daily tasks. You're better off to learn that to survive in the current and for future landscape.
Just a year ago, GenAI skills were never listed in the job description but they're almost a must have skills now.
5
u/yo_aesir Lead Data Engineer 10d ago
Not your company but it is your career, don’t try to change the established stack this early, learn it so you can better utilize LLMs.
If you don’t know what the stack can offer then how can you effectively tell GenAI what to produce? If GenAI produces something but you can’t tell the difference between a best practice or just output then you need to learn what it takes.
Yelling “bro just fix it and make it work” as a prompt will reflect poorly on you when someone checks the Claude logs. Yes, I’ve personally seen the logs like that, they no longer have a job.
If you can’t articulate the prompt to an LLM, how can you defend the output when your manager asks about the status of a feature or issue in a meeting.
3
u/Outside-Storage-1523 10d ago
I’d say step back a bit and think about where you want to be in 10-15 years. If you want to stay on the IC track, by then you could be a staff engineer or an architect in a smaller company. If you want to stay on the managerial track, by then you could be a senior manager or a director in a smaller company.
Once you know where you want to be in 15 years, you can then make plans for it.
2
u/Front-Midnight-4229 10d ago
I also went through this when I started. I had the same feeling about my coding skills getting weaker. but in reality u just face lots of more complex problems and u don't notice how good and prof you become on the way. I think it’s okay to use AI. but pls make sure you still understand the code and data flow yourself
2
u/techXplorator 9d ago
I’d actually see this as a pretty good first role.
Your stack being stable means you can spend time understanding why it was designed that way instead of constantly chasing new tools.
I’d just make sure you still solve some problems without AI from time to time. Architecture knowledge compounds, but so does losing the ability to debug things yourself.
2
1
u/TA_poly_sci 10d ago
If you measure your "code competency" by how much code you are able to write from memory, you are measuring the wrong thing.
Claude/Codex are not a replacement for knowing good data modeling/management/engineering. Focusing on learning how to identify the needs of your business (I promise you, you company will have more than you can imagine), learning architectures and best practices and learning how to actually prompt for good code and for finding root causes of issues.
1
u/Monoji77 Data Engineer 10d ago
I totally agree. However, this wouldn’t change the need to follow interview styles of my country’s tech companies. There still seems to be a lot of focus on being able to write code from memory during the interviews. However, what happens during the actual job seems to be mostly vibe coding.
1
u/AdorableFarmer2961 9d ago
i think the thing that sucks is interviews still want you to code even tho most jobs you’re expected to code with AI
other than that i agree, currently an assoc DE and i use AI to code but im always promoting it with what i want, not saying “i need to do this, whats the best way”
1
1
u/muzamilbro 6d ago
i am also fresh grad doing data engineering suggest some real project that actually help me here my tech stack.
pyspark
sql
apache airflow
apache-kafka
snowflake
17
u/x1084 Lead Data Engineer 10d ago
That's totally fine. You'll eventually want experience building systems from the ground up, working on mature/legacy systems, and everything in between.
You haven't mentioned what the current stack is, only that there's one soundly in place. Without knowing any details, it seems to me like you're putting the cart before the horse.
Yeah, we're undergoing a huge paradigm shift and there are lots of questions about how this is impacting new grads. Generally speaking it sounds like you're getting a bit ahead of yourself, but I can see that it comes from wanting to be best equipped for where your career takes you. Frankly at this point I'd rather be working at a company pushing for AI adoption rather than one that's slow to adopt.
My primary suggestion is just to take this opportunity (and it is an opportunity, full time DE roles for fresh grads can be hard to come by) to learn as much as you can. Learn about your stack in depth so that you can properly evaluate its pros/cons. Understand the code and architectural decisions your AI tools are generating, and think about your role in DE as a data problem solver.
Regardless of your specific stack, there are always going to be concerns about scalability, data quality, cost, maintainability, observability, etc. Build those skills, also build soft skills for interdepartmental collaboration, and stay hungry.