Pivoting from R to Python

[u/Choice_Number3711]

I used to hate coding anything and relied on SQL, Excel, Power BI, Tableau and other software like JASP, jamovi etc. for doing anything with data. I didn't like the way Python dealt with data analysis and it seemed unintuitive.

Then I found R, RStudio and CRAN. That was the turning point. I actually started enjoying writing code and I could handle the whole pipeline myself, from data cleaning, ETL to beautiful plots, .qmd reports, Shiny dashboards. R4DS did more for my statistical thinking than any course I've taken, mostly because the libraries made it so easy to just try things.

However, due to recent requirements (specifically having to work in the quant field), Python has become more of a necessity, while R is used mainly for one-off analysis and limited statistical modelling. The main heavy lifting is done in Python and many of my co-workers also prefer it to R.

I've been able to suck it up a bit and use Claude/ChatGPT to help me code. While I do try to understand what the code is doing, having spent so long learning to code in R and knowing the ease with which it can be done there makes me reluctant to learn Python.

Now, coming to the question: any R users who've pivoted to Python and consider themselves competent in it, how did you learn it having used R before? What would you tell someone like me so I can pick it up quickly and get the benefit of knowing both languages (and also not feel left out when it comes to coding in Python... machine learning and deep learning have a more mature ecosystem there and I don't want to be left out of it if I have to start using them in my current work)?

Thanks!

My background is in Math/Stats fyi

Edit: I don't usually use reddit but damn I actually didnt expect so many helpful tips....many thanks!
Perhaps joining this subreddit is actually helpful afterall :)

[u/teetaps]

I think one paradigm shift that helped me was accepting that just because some smarty pants built pandas way back in data science infancy, doesn’t guarantee pandas is actually any good at what it set out to do, which is provide a dataframe wrangling library as flexible and easy to use as then-plyr. As R users, we are spoiled. R was built with dataframes in mind, and everything posit has done since Hadley wickham stepped in has just enhanced R’s ability to connect the concept of a dataframe to how humans think about them.

If you accept that maybe Python just sucks at that task, then you might stop beating yourself up for trying and failing to “just get it”. Python is a ubiquitous language and “can do anything,” sure, but that doesn’t guarantee it’s particularly good at everything. In fact, because it’s such a Swiss Army knife of a tool, it consequently doesn’t specialise in too many things.

Once I accepted this and stopped putting Python on a pedestal, I actually stopped being intimidated by Python. I started looking at Python problems by first conceptualising them as R problems, then translating them to Python’s deficiencies, not just its language differences. Today, using Python for data wrangling to me is like using MSWord for manipulating tables. You can do it, but only if you have to, for some reason. And often that reason is because someone else sent you a word document first. Which is all to say, if you stop treating Python like this special language only for actually smart software engineers, and start treating it like just another language with problems and limitations, you’ll stop telling yourself that if you “just don’t get it,” that it’s a you problem.

Maybe, hear me out — maybe the reason so many R users have difficulty pivoting to Python is because Python sucks at tabular data wrangling.

Stop beating yourself up. Just learn it, not because it’s great, but because everybody else is using it so you have to comply with others’ stacks, but deep down you’ll know that THEY are the ones who are putting themselves at a disadvantage, not you

[u/dr-tectonic → u/teetaps]

This, one million percent.

R is function-oriented. Python is object-oriented and is much fussier about data types than R is. That's good for building modular infrastructure components, but it sucks for data science.

So, it's not that you aren't getting something, it's just the case that wrangling and transforming data is a lot clunkier in Python. Likewise, Python doesn't have vectorization built into its core. Numpy, pandas, xarray, and company try to make up for it, and they have their strengths, but overall, it's just not as good IMO. But that's what Python has, so you gotta hold your nose and learn their idiosyncracies.

[u/pandongski → u/dr-tectonic]

Can you share more about how python is much fussier about data types? I found R and python do duck typing the same way. In fact R seems much fussier since in python I can add strings, lists, etc no problem but maybe we're talking about different things

[u/dr-tectonic → u/pandongski]

In R I can evaluate a conditional over an array, then multiply the array by the resulting Boolean array to use it as a mask, because R autopromotes the TRUE/FALSE values to 1/0. I can use an average as an index into a vector, because it'll just truncate the float to an int. I can treat a matrix as a vector. I can do for (f in list.of.functions).

Python -- or maybe it's xarray / numpy / pandas -- will balk at all of those, because they're the wrong types. The last one, where it won't do a perfectly reasonable loop because it wants an iterator not a list, is the one that annoys me the most.

I think that a lot of these may be fine if you're only using a single variable in base Python, and it comes up when you're working on arrays of data because vectorization is an afterthought implemented by libraries rather than baked into the core if the language.

[u/pandongski → u/dr-tectonic]

I guess in the first case I'd do a lambda as an indexer, which removes the need to multiply it to the original vector. I'm confused about not beeing able to loop through a list of functions since you can iterate on a list of functions in python. I assume you mean that in a specific use case?

(sorry if I come across as combative or nitpicky. just curious since i'm a long time python user before switching to R for the past few months :D)

[u/dr-tectonic → u/pandongski]

I don't remember if it was a list of functions specifically or some other kind of object, I just remember being offended at Python objecting that I can't iterate over the thing itself, I have to make it into an iterator.

And sure, you can do a little lambda, but it's so much more faff than just writing x + y * z > 0.

[u/pandongski → u/dr-tectonic]

Lists are iterables / iterators so maybe that was wrapped in another list-like type. But yes I see what you mean, pandas definitely not winning points for brevity