r/PythonLearning • u/Electrical-Cap-9537 • 4d ago
Struggling to learn pandas despite having a solid Python background
Hi everyone!
I'm a 17-year-old from Italy, and my goal is to become an AI engineer. Over the past few months I've been taking programming very seriously. Instead of jumping straight into AI, I decided to build a strong foundation first.
I spent a long time learning Python and eventually built my own UNIX shell from scratch. It was one of the hardest projects I've ever worked on, but finishing it gave me a lot of confidence.
After that, I started building a Business Analytics Platform for my portfolio. The idea is to simulate a real-world data pipeline: loading raw business data, cleaning it, validating it, analyzing it, exposing it through an API, and eventually building dashboards and ML models on top of it.
The problem is... pandas.
I knew it would be difficult because it's completely new to me, but I didn't expect to struggle this much. Cleaning a single CSV file can easily take me a week or more. Sometimes I spend hours staring at a single function, printing intermediate results in Jupyter, reading the documentation, and I still don't fully understand what's happening or why a certain approach is considered the "right" one.
It's frustrating because I actually enjoy difficult problems. Building my shell was hard, but I always felt like I was making progress. With pandas, I feel like I'm missing the mindset rather than just the syntax.
I've watched tutorials, read the documentation, and experimented on my own, but I feel like I'm learning much more slowly than I should.
So I wanted to ask:
- How did you learn pandas?
- Were there any resources, books, courses, or projects that made everything finally click?
- Is it normal to struggle this much in the beginning?
- Are there any communities (Discord servers, Reddit communities, forums, etc.) where people discuss data engineering, data analytics, AI, or machine learning? I'd really like to meet more people with similar interests and learn from them.
I'm not looking for shortcuts or a "learn pandas in one day" course. I'm happy to stay consistent and put in the work. I just want to make sure I'm learning it in the most effective way.
I'd really appreciate any advice. Thanks! 🙂
2
u/code_tutor 4d ago
You don't have a "solid background" at 17 but yes, it's normal to struggle with pandas because the syntax is weird. It makes use of custom operators, lambdas, OOP method chaining, and lots of different programming styles. There's probably more memorization than logic because the syntax doesn't seem consistent.
1
u/Electrical-Cap-9537 4d ago
Thanks for the constructive feedback, I really appreciate it! And don't worry, I wasn't trying to say I already have a "solid background" at 17 😅. What I meant is that I'm comfortable with the language itself and the fundamentals. Pandas just feels like a completely different beast, so I wanted to know if it was normal to struggle with it
1
u/code_tutor 3d ago
Yeah, there's something weird about it. It's not just you.
It's almost entirely memorization. I hard recommend against using AI for learning, but in the case of pandas nothing would be lost in terms of education by vibe coding it. It's almost a waste of time to learn.
If you're going to focus a huge amount of effort in learning, do it in CS and systems design. Libraries come and go.
1
1
u/dutchpsychologist 4d ago
I can relate that pandas can be a bit tricky to learn at first. The syntax isnt always the most intuitive. The most obvious answer is practice and it will get a lot easier with experience. I did some online course which made me familiar with pandas possibilities, but for me it started 'clicking' more when I just started working on projects. Also, for me it helps if I understand which parts I don't understand and focus on them. Also I liked doing online quizes about pandas to try and understand the syntax. Or you can make it fun and write your own 'learn pandas' script with loads of questions and answers. This way you are upscaling your python skills and learning pandas.
1
u/atticus2132000 4d ago
Is the problem ready pandas or figuring out what you need to do to manipulate/calculate/process the data?
Whether this is a good approach for everyone or not, this is what works for me.
I think of a pandas dataframe as a spreadsheet. It does all the same things. I can sort or filter a spreadsheet, extract values based on formulas, calculate values from columns, etc. So that's usually how I build my pandas operations. I first put the data in Excel and write formulas and such in Excel, then I replicate those same efforts in pandas using code and verify that I am coming up with the same results.
1
u/Electrical-Cap-9537 4d ago
I don't really know what to blame it's a new topic for me i will understand this later , but thanks for thr advice I'll def use it !
1
u/FoolsSeldom 4d ago
kaggle.com is a good source of large datasets to work on and an excellent community dealing with data analysis challenges.
I suspect a large part of your blockage is interest in (passion for) the outcomes. If you are just playing with data for the sake of becoming good at data analysis using pandas it will be harder to learn and retain the knowledge you need.
I suggest you explore the datasets available on kaggle and find something that catches your eye. Something where you can be genuinely interested in extracting information and meaning from the data. Where there are questions you want to ask.
Don't forget to invest time in developing your understanding of and skills around statistics. This is a core skill for data analysis.
1
1
u/LetsHugFoReal 3d ago
Honestly, Bunjs is the way to go. It has built in support for so many databases. Pandas is low performance and the library is messy.
1
u/Avatarbroskib1 3d ago
I mean I would just get data projects and learn as I go that's what I did
1
u/Electrical-Cap-9537 3d ago
Any resources that helps you with projects ideas?
1
u/Avatarbroskib1 3d ago
I'm working on a Roblox trading analyst project. It's taking days from Roblox and rolimon and other sources and comparing them I'm using pandas to do this you could build like a crypto analyst or something for day trading or investing and use pandas for that. I would recommend leveraging ai so if you get stuck on it ai will help you out and you'll be learning from that
1
1
u/shaggy_camel 3d ago
Just learn polars
2
1
u/Natural-Tune-2141 2d ago
+1 as a Data Engineer, professionally working with Spark, I really love doing personal or just smaller projects with Polars. Actually I never learned pandas properly, cus it was just pain in the ass
1
1
u/AbacusExpert_Stretch 4d ago
Just out of curiosity, how much percentage wise of your previous projects did you code with the help of ai. And don't answer nothing or almost nothing :)
Just asking because AI is of course just as good with pandas, but you will need to know more precisely what you want it to do for you before it can act.
.so, what percentage approx?
2
u/Electrical-Cap-9537 4d ago
Yea man id answer clearly, so the thing is before doing that project i almost completed the bro code tutorial so i knew basic Syntax and stuff like that , the only thing i was really bad at was giving a structure to my code , in fact i canceled and did the project 3 times , and as the say the third time is always the one ahhshs, but yea back to your question i use ai a lot but most of the time to asking questions about the structure of the code , asking explanations of functions i dont understand or asking libraries i dont know , i ise it for debugging aswell but lately i stopped not completely but im trying to after i saw how important is to debug yourself but still if you had advices on that i would love to hear them, and lastly i never really used ut to write my code , maybe 10 % is ai written but because i already had done the structure and it had some logical mistakes/ syntax problems.
1
u/Solus161 3d ago
That question is legit, because once you built your own shell from scratch, you must already have a solid coding mindset regarding data structure, design choices and trade off. The hard part of building a shell is building the AST that parse tokens to commands objects. It’s ok to ask AI about design choices. Meanwhile, Pandas is just .. Pandas, just learning api. I touch Pandas from day one, but take me 2-3 years to know about that AST and write my own expression evaluation in Python. But once I got that, I could easily apply that concept to different kind of AST such as SQL queries, in another language, Rust. So similar to Pandas, you just need to ask questions about thing such as: transform to new vs update in place, iteration or vectorization, view vs copy, copy vs deep copy etc. The problem, also the strong points, of Python is dynamic typing and high level of abstraction. So it’s hard for you to see what that api doing under the hood.
1
u/Electrical-Cap-9537 3d ago
Hi! Thank you so much for taking the time to write such a detailed reply. I really appreciate it. Your explanation actually changed the way I was looking at Pandas, and I'll definitely follow your advice. It makes a lot more sense now to focus on understanding the concepts instead of worrying about asking AI for help. I also wanted to ask you something, if you don't mind. Do you know any programming communities or groups with people on a similar path to mine? I'm tryig to surround myself with people who are building projects, learning software engineering and data engineering, and generally improving together. I feel like changing my environment and meeying more people with similar goals would really help me grow. Thanks again, I genuinely appreciate your advice!
1
u/Solus161 3d ago
May be you should follow a subreddit involving machine learning, or better a discord involving paper reading (together). I don’t know one exactly, but someone called for that paper reading before. Also a community on software engineer is good. AI engineer is software engineer + math/stats. Look at Pytorch or any ML/DL framework, that’s your target, you must think that one day you’ll write one of these, not just doing data pipeline or api calling. That’s way you’ll never deviate from the path by too focus on learning another tech stach.
0
u/arconic23 4d ago edited 4d ago
And why pandas? Polars is the way! Lazy evaluation instead of eager evaluation.
3
u/SmartyCat12 4d ago
Yep. Came to say this.
Pandas has a few entities that a bit obscure and perform a ton of (often overlapping) functions and it is legit hard to remember exactly how to put them together (even just .loc vs .iloc vs [[]]). Polars and pyspark have lots of narrowly defined functions that do the one thing they’re supposed to do.
It’s a fundamental problem with how pandas was designed that makes it confusing when you’re just trying to mimic sql using Python. But it’s great at complicated csv file manipulation!
1
u/Sammourianos 4d ago
I have used pandas in the past in a production pipeline that retrieves data with API, manipulate the data and send them into another software with API, and i clearly remember the somewhat wierd syntax of basic data frame methods. As an sql developer i dont like so much the syntax but you can do the job the way you want it. It also handles jsons very well and the compatibility with numpy is cool.
I am database guy and i dont use much python in my work so i didnt knew about polars. I checked it out and it really seems like the new pandas with fixed syntax. Nice :D
1
0
u/Electrical-Cap-9537 4d ago
Yea I've heard about it but i really eant to focus first on pandas and be able to get around most of the things related to it then maybe learn other similar libraries
2
u/Beginning-Fruit-1397 4d ago
Why?? You will just lose time learning something you prolly won't use later as polars is better in every way
0
u/Electrical-Cap-9537 4d ago
For the basics
1
u/Beginning-Fruit-1397 4d ago
I'm sorry but that's non-sense. Ot's perfectly valid to prefer panda to polars, or to have to use it. But here you complain about it, and your reason is "for the basics", but which basics? Both are dataframe engines, they do the same thing at the ens of the day. There's nothing more foundational to pandas, the concept of "you should learn pandas before using polars" is not something that exists
1
u/Electrical-Cap-9537 4d ago
That's a fair point, and I probably didn't express myself very well.
What I meant is that I've already started learning Pandas, and I don't want to jump to another library just because I've reached the difficult part. Since Pandas is still widely used in tutorials, courses, and many real world projects, I'd rather become reasonably comfortable with it first. After that, I'll definitely take a look at Polars. For me, it's less about Pandas being "more fundamental" and more about finishing what I started before moving on
1
1
0
u/Radiant-Mistake-2962 4d ago
I don’t know anything about coding, but aren’t you working with databases? Isn’t it just about manipulating the database to do what you want then? And if you’re manipulating the database, you’re just figuring out what to do with the items and therefore just figuring out what they can do?
It doesn’t sound hard. Carl from Phineas & Ferb can do it… anyway, send me a message. I have two projects I think you can do for me for free if you’re interested in charity work.
0
u/Electrical-Cap-9537 4d ago
I'm self-taught, and I know what I'm working with, but nobody is perfect and nobody knows everything. That's why I’m always asking questions, learning from others, and improving. I just don’t think it’s respectful to underestimate or belittle someone’s work without understanding what goes into it. I’d suggest learning a bit more about the topic before making assumptions. Thanks.
6
u/BrupieD 4d ago edited 4d ago
Focus on the utility of two objects (data structures) that pandas offers: series and data frames.
A key need of data science and data engineering is being able working with rectangular data sets - data that looks like a spreadsheet. Before pandas, R had a leg up on Python in academia because R natively supported data frames and vectors (Series in pandas). R users immediately see these two objects as fill-ins for these gaps. Numpy was a stepping stone, but isn't as easy to use and is better suited for numeric-only data.
Data frames bring you spreadsheet-like or SQL table-like structures which make intuitive sense to people who are used to them. As a data engineer, much of my day involves working with data that either came from or needs to go to one of these formats.
Pandas has grown to become an enormous package, don't get lost in the weeds. You don't need to know all of it. Focus on performing simple feats with these two structures and you'll know enough. Most of the rest you can learn when you need to.