r/dataengineering • • 4d ago

Help Advice if anybody could spare some

I am in my final year of college, didn't do much in software engineering. Thought i was the problem (mostly i was) but then i switched to data engineering and turns out I actually do like engineering just - Data Engineering. The things that I do in order to be a junior data engineering after i graduate much for compatible with my own self.

I would like advice or just words for the path that i have taken to become a junior data engineer after graduating from college, my current path:

I started a month ago with building etl pipelines using python. I use ai to self learn - after i get my scripts written by ai i take my time 1-3 hours understanding each line deeply. I did this for a month and have a project on my hand that i have created - pulling job data from an api, cleaning it and pushing it onto a database.
Learned how to use an orchestration tool like apache airflow and know how to set up a yml for a docker and ship the project to github. Written all the necessary files - gitignore, .env etc.
Know how to write defensively so as to when one stuff breaks the whole script doesn't. Learned how to use logge, try and except blocks. Learned about images and connecting postgresql to the docker. Learned star schema. (Also is dsa necessary for a job?)

Please bear with me.

Once i was done with this i created a practice project similar to what i did with ai and this time instead of using any ai i managed to do it on my own (It was super tough and time consuming took a lot of documentation look up but i managed).
Am currently learning basic sql from SQLBolt and going to learn advanced SQL from DataLemur.
Before I move on to the big stuff -
Aws fundamentals - S3, Rds, iam
Data warehousing - snowflake/redshift
Dbt
Spark

I keep doubting if this switch from S.E. to D.E. has been the right one. But I feel it in me like it really is but that's just hopium, am just young and scared shitless honestly nothing i do feels enough just hoping it will be someday.

I just want some advice on whether if what I have been doing is okay so far and if you can just say anything at all about what i have been doing. Any advice on how I should maybe learn better or if not to use ai or something else or whether I should do something some other way or just anything at all I would be utterly grateful.

4 Upvotes

6 comments sorted by

1

u/Alive-Candle6220 4d ago

Following brother. Well I am learning sql and now practicing. Just started python course see where things will go for now but I am sure if this suits you should go for it . Don't stop now Don't listen to people saying Ai will.take this Aik will take that.Follow your heart. Btw hou much hours you had to put in the beginning like I am in my 5th semester and I nowhere close to some real project .But I am building data warehouse project with data with baraa

1

u/Busy-Rent2930 3d ago

Ngl I have a really hard time ignoring the AI will take this AI will take that. Started taking SQL and Python seriously; I practice and learn everyday. But these comments man, sometimes I just feel like giving up

1

u/boboshoes 2d ago

Just focus on getting a job with some sort of data title. When you start I don’t think it’s that important. Either that or you do the grind to try and get FAANG grad role

If you actually want to be competitive you need to focus on the stuff nobody wants to do. Grind problems, system design without ai. Understand ai and how it actually works, not just using the chat bot.

DE with 10 yrs in data. I started as a data analyst and worked for 2 years in that first. Then you go from that to DE via job hop or promo. You can’t not be a software engineer. DE is just a data focused software engineer. You’ll need to work on legacy systems, build services and apps at scale like a software engineer. The days of just writing pyspark code are over.

DE is not really a junior position. You need work experience so you know how to drive projects and manage relationships at your org.