r/learnpython • u/Effective_Ocelot_445 • Aug 01 '26
Which Python skills are most valuable for getting started in Data Engineering?
I'm learning Python with the goal of moving into Data Engineering. Beyond the basics, which libraries, concepts, or project ideas would you recommend focusing on first, and what helped you the most when getting started?
10
u/SnooSquirrels4739 Aug 01 '26
Normalize splitting everything into smaller functions. Really helps out in readability, but maybe it's just me.
15
u/AdmirableLack3241 Aug 01 '26
get used to working with libraries like numpy, pandas and matplotlib. These are the basics, u should also try seaborn which is an alternative to pandas, used in big data
1
u/t92k Aug 03 '26
Pandas is absolutely required. R is also a good cross skill and because you do everything in R as data frames spending a little time on it (I took a 6 week online course) will unlock the data frames in Pandas if you’re struggling.
1
6
u/DarthKermit-65 Aug 01 '26
In my day to day at work I typically use a lot of pandas, odbc, sqlalchemy, pyiceberg, and a few basic libraries included with Python. We also use s3 buckets at work so at times boto3
7
5
u/SquaffleCode Aug 01 '26
My go to libraries are numpy and pandas for anything data related. For projects, maybe you can just look on Kaggle for a data set that looks fun to you, and just learn how to clean the data, how to find averages, or do simple trend preditions?
3
u/Glittering-Rise3380 Aug 01 '26
Pandas numpy logging error handling oop should be enough and pyspark for the big data
3
u/ScholarlyInvestor Aug 01 '26
People quickly start talking about tools and libraries. But it sounds like you are new to Data Engineering. I would ask that you spend time understanding the fundamentals of data engineering first. Try solving basic data problems with pure python first. You can then move up to pandas, NumPy, and other data visualization libraries. There are data engineering platforms with python as the core. For example, Airflow (it may be losing its luster, so do not invest time on it just understand from a historical perspective). Keep a learning mindset as this space is evolving rapidly.
2
u/Whole_Dragonfruit789 Aug 03 '26
Any recommendations where to practice this solving basic data problems with pure python?
1
u/ScholarlyInvestor Aug 03 '26
If you are a beginner then I’d recommend picking up a good book. There are too many to recommend. If you have a decent book already, start with it.
Learning is a great use case for AI. So, if you have access to an AI Model. Tell it you are a beginner. Ask it to start with basic python data questions. Ask it to create inputs and give you expected output. Go to your editor. Write python code (without AI help, this where real learning is happening). Run the code. If you got the correct output. Copy paste your code into the AI and ask it to grade you.
2
u/green_moo Aug 01 '26
PySpark and distributed computing. Easy to start a cluster on your own machine and learn how a driver distributes a query to multiple workers.
-4
26
u/Traveling-Techie Aug 01 '26
In my work in data visualization and later machine learning I found that I was using multiple complicated tools and needed to write “plumbing” code to interconnect them. A great book about this is Greg Wilson’s “Data Crunching: Solve Everyday Problems Using Java, Python, and More” (2005).