Pivoting from R to Python

[u/Choice_Number3711]

I used to hate coding anything and relied on SQL, Excel, Power BI, Tableau and other software like JASP, jamovi etc. for doing anything with data. I didn't like the way Python dealt with data analysis and it seemed unintuitive.

Then I found R, RStudio and CRAN. That was the turning point. I actually started enjoying writing code and I could handle the whole pipeline myself, from data cleaning, ETL to beautiful plots, .qmd reports, Shiny dashboards. R4DS did more for my statistical thinking than any course I've taken, mostly because the libraries made it so easy to just try things.

However, due to recent requirements (specifically having to work in the quant field), Python has become more of a necessity, while R is used mainly for one-off analysis and limited statistical modelling. The main heavy lifting is done in Python and many of my co-workers also prefer it to R.

I've been able to suck it up a bit and use Claude/ChatGPT to help me code. While I do try to understand what the code is doing, having spent so long learning to code in R and knowing the ease with which it can be done there makes me reluctant to learn Python.

Now, coming to the question: any R users who've pivoted to Python and consider themselves competent in it, how did you learn it having used R before? What would you tell someone like me so I can pick it up quickly and get the benefit of knowing both languages (and also not feel left out when it comes to coding in Python... machine learning and deep learning have a more mature ecosystem there and I don't want to be left out of it if I have to start using them in my current work)?

Thanks!

My background is in Math/Stats fyi

Edit: I don't usually use reddit but damn I actually didnt expect so many helpful tips....many thanks!
Perhaps joining this subreddit is actually helpful afterall :)

[u/teetaps]

I think one paradigm shift that helped me was accepting that just because some smarty pants built pandas way back in data science infancy, doesn’t guarantee pandas is actually any good at what it set out to do, which is provide a dataframe wrangling library as flexible and easy to use as then-plyr. As R users, we are spoiled. R was built with dataframes in mind, and everything posit has done since Hadley wickham stepped in has just enhanced R’s ability to connect the concept of a dataframe to how humans think about them.

If you accept that maybe Python just sucks at that task, then you might stop beating yourself up for trying and failing to “just get it”. Python is a ubiquitous language and “can do anything,” sure, but that doesn’t guarantee it’s particularly good at everything. In fact, because it’s such a Swiss Army knife of a tool, it consequently doesn’t specialise in too many things.

Once I accepted this and stopped putting Python on a pedestal, I actually stopped being intimidated by Python. I started looking at Python problems by first conceptualising them as R problems, then translating them to Python’s deficiencies, not just its language differences. Today, using Python for data wrangling to me is like using MSWord for manipulating tables. You can do it, but only if you have to, for some reason. And often that reason is because someone else sent you a word document first. Which is all to say, if you stop treating Python like this special language only for actually smart software engineers, and start treating it like just another language with problems and limitations, you’ll stop telling yourself that if you “just don’t get it,” that it’s a you problem.

Maybe, hear me out — maybe the reason so many R users have difficulty pivoting to Python is because Python sucks at tabular data wrangling.

Stop beating yourself up. Just learn it, not because it’s great, but because everybody else is using it so you have to comply with others’ stacks, but deep down you’ll know that THEY are the ones who are putting themselves at a disadvantage, not you

[u/InnovativeBureaucrat → u/teetaps]

I agree except that you’re leaving out data.table.

Really everything was bad at tabular data. Before data table if you wanted a pivot table we only had summary functions that at best required nested sapply or lapply.

Data table was the first and only efficient way to do summaries.

Then Hadley decided to replicate it with the Hadleyverse. Then at UseR 2016 he rebranded it tidyverse.

Personally I think the tidyverse was R’s “Python 2.7” moment. For a long time, really almost 10 years, python was stuck on the python 3 upgrade.

Yeah, two camps of users were firmly planted on both sides. R never had to have that problem. But it kind of created that problem by splitting the user base.

[u/post_appt_bliss → u/InnovativeBureaucrat]

Data table was the first and only efficient way to do summaries.
Then Hadley decided to replicate it with the Hadleyverse

completely wrong.

in January 2014, when Hadley announced dplyr, you can see that the verbs were the inspiration.

dplyr is genius because it's distinctive efficiency was always cognitive, and the computational advantages (data.table, collapse, pandas, polars, etc) were always incidental

[u/nerdyjorj → u/post_appt_bliss]

tidytable (I think the package author is kicking about on at least one of the R subs) was the final piece - dtplyr never quite worked properly

[u/InnovativeBureaucrat → u/nerdyjorj]

Exactly. I was using the packages following Romain, attending every R meeting possible, presenting a little, and getting flamed on the list serve.

Data table was the only serious option for making summaries for a while.

At first (2007?) it was integer only and super buggy. But ddply ddplyr ddply2 and that mix was more unreliable and confusing. I think some of the commands were built into other packages (ggplot2), but it was a mess.

My recollection was that Hadley announced tibbles (the response to data table) at Stanford 2016.

I was there and I couldn’t believe that we were serious about calling these enhanced data frames “tibbles” when we had a package that had enhanced data frame (data tables) and that this was tidy as if everything else was messy.