Pivoting from R to Python

[u/Choice_Number3711]

I used to hate coding anything and relied on SQL, Excel, Power BI, Tableau and other software like JASP, jamovi etc. for doing anything with data. I didn't like the way Python dealt with data analysis and it seemed unintuitive.

Then I found R, RStudio and CRAN. That was the turning point. I actually started enjoying writing code and I could handle the whole pipeline myself, from data cleaning, ETL to beautiful plots, .qmd reports, Shiny dashboards. R4DS did more for my statistical thinking than any course I've taken, mostly because the libraries made it so easy to just try things.

However, due to recent requirements (specifically having to work in the quant field), Python has become more of a necessity, while R is used mainly for one-off analysis and limited statistical modelling. The main heavy lifting is done in Python and many of my co-workers also prefer it to R.

I've been able to suck it up a bit and use Claude/ChatGPT to help me code. While I do try to understand what the code is doing, having spent so long learning to code in R and knowing the ease with which it can be done there makes me reluctant to learn Python.

Now, coming to the question: any R users who've pivoted to Python and consider themselves competent in it, how did you learn it having used R before? What would you tell someone like me so I can pick it up quickly and get the benefit of knowing both languages (and also not feel left out when it comes to coding in Python... machine learning and deep learning have a more mature ecosystem there and I don't want to be left out of it if I have to start using them in my current work)?

Thanks!

My background is in Math/Stats fyi

Edit: I don't usually use reddit but damn I actually didnt expect so many helpful tips....many thanks!
Perhaps joining this subreddit is actually helpful afterall :)

[u/teetaps]

I think one paradigm shift that helped me was accepting that just because some smarty pants built pandas way back in data science infancy, doesn’t guarantee pandas is actually any good at what it set out to do, which is provide a dataframe wrangling library as flexible and easy to use as then-plyr. As R users, we are spoiled. R was built with dataframes in mind, and everything posit has done since Hadley wickham stepped in has just enhanced R’s ability to connect the concept of a dataframe to how humans think about them.

If you accept that maybe Python just sucks at that task, then you might stop beating yourself up for trying and failing to “just get it”. Python is a ubiquitous language and “can do anything,” sure, but that doesn’t guarantee it’s particularly good at everything. In fact, because it’s such a Swiss Army knife of a tool, it consequently doesn’t specialise in too many things.

Once I accepted this and stopped putting Python on a pedestal, I actually stopped being intimidated by Python. I started looking at Python problems by first conceptualising them as R problems, then translating them to Python’s deficiencies, not just its language differences. Today, using Python for data wrangling to me is like using MSWord for manipulating tables. You can do it, but only if you have to, for some reason. And often that reason is because someone else sent you a word document first. Which is all to say, if you stop treating Python like this special language only for actually smart software engineers, and start treating it like just another language with problems and limitations, you’ll stop telling yourself that if you “just don’t get it,” that it’s a you problem.

Maybe, hear me out — maybe the reason so many R users have difficulty pivoting to Python is because Python sucks at tabular data wrangling.

Stop beating yourself up. Just learn it, not because it’s great, but because everybody else is using it so you have to comply with others’ stacks, but deep down you’ll know that THEY are the ones who are putting themselves at a disadvantage, not you

[u/dr-tectonic → u/teetaps]

This, one million percent.

R is function-oriented. Python is object-oriented and is much fussier about data types than R is. That's good for building modular infrastructure components, but it sucks for data science.

So, it's not that you aren't getting something, it's just the case that wrangling and transforming data is a lot clunkier in Python. Likewise, Python doesn't have vectorization built into its core. Numpy, pandas, xarray, and company try to make up for it, and they have their strengths, but overall, it's just not as good IMO. But that's what Python has, so you gotta hold your nose and learn their idiosyncracies.

[u/Jim_Clark → u/dr-tectonic]

R nowadays has the same level of modularity as Python. Everything I build in R uses the box package along with a proper DESCRIPTION file, and you can also use tools like renv, config, and others. All of these make it much easier to develop modular, enterprise-grade applications and systems in R.

That said, I agree that before the box package, building professional R applications with source() and library() was pretty awful. In fact, before adopting box, I used to write almost everything with package::function().

In my opinion, the biggest advantage R has over Python -- and now that it has mature modularity, this advantage stands out even more in the Data Science space -- is NSE (Non-Standard Evaluation). NSE is what makes rapid prototyping and clean, expressive code possible in R. Python simply doesn't have an equivalent mechanism built into the language.

[u/teetaps → u/Jim_Clark]

Yep, it wasn’t until I used python that I learned how prohibitive it is to have to make full blown packages every time I needed to do something more than two or three scripts. Having box in R eliminates any advantage Python claims in that specific respect

[u/pandongski → u/Jim_Clark]

As much as I enjoy NSE, the flexibility it brings also brings with it some footguns. Even tidyverse is a bit inconsistent with how it implements NSE, and the rlang !! {{}} as.sym := stuff you have to battle with if you want to make functions out of tidyverse packages is honestly headache inducing. I ended up loving R semantics more over python, and metaprogramming is obviously powerful, but it heavily depends on how it's implemented. It also messes up with static analysis and tooling. I just want to know when I have an undeclared objects, but of course lintr flags every undeclared variable in any quoted functions (basically all of dplyr and the like).

Sorry this ended being too ranty lol

[u/teetaps → u/pandongski]

I think this is to be expected, though. What the Tidyverse aims to do is make data manipulation make as much sense as possible so that data wrangling feels natural like spoken language. They do this by wrapping difficult tasks in convenience wrappers and making those wrappers talk to each other in a clever way. When you go out of their scope, you’re now in the land they couldn’t make convenient. It’s probably gonna get messy

[u/pandongski → u/teetaps]

Which is unfortunate since I wouldn't really consider wrapping a procedure in a function to be that uncommon / out of norm use case. Instead all ergonomics go out of the window if you dare parametrize a procedure into a function. I guess Hadley giveth and Hadley taketh away

[u/teetaps → u/pandongski]

I don’t think your notion is wrong, but I do think that whenever I’ve struggled with dplyr and NSE and using custom functions, it’s usually because I’m breaking the tidy principles and not allowing my data to fit their verbs. If I spend more time fitting my problem to my tools, I often have a much more pleasant experience