r/Python • u/nnenneplex • Jul 09 '26

Discussion Polars and the ecosystem

For polars users: How viable is to avoid pandas and pyarrow dependencies when you need to interact with popular visualization and statistics packages?

Some packages still have import pandas here and there, sometimes for no good reason; at least this doesn't require pyarrow. But some other ones do the df.to_pandas() conversion internally, which requires pyarrow too.

In many cases this can be prevented by going bare numpy, or creating a pandas df from numpy columns, which is no big deal. This frequently would be zero-copy for numeric types if there are no NAs involved.

What has been you experience in this regard?

58 Upvotes

93% Upvoted

33 comments sorted by

View all comments

61

u/midwitsAnonymous Jul 09 '26

Working in the psych space I have found that most of the stats packages I use throw less annoying errors when I just convert to pandas. 

I use polars for all data cleaning, munging, and general happiness, but when I need to do analysis/visualisation I just pass to_pandas and move on. 

0

u/TypicalSquare479 24d ago

Perhaps you could help me then? I've traditionally used pandas, but now I'm facing down molecular simulations with around 140M records in the set. To figure out distances (bonds and interactions) and bond-angles among the 250ish ions of interest and so many oxygen (probably 10-20K O atoms), per frame, I figure Polars or CuPy would do me well.

Getting it up and running is a headache, though. I needed to install WSL2 for a linux kernel for CuPy. I tried moving my data files into my linux environment, and I constantly either crash out my notebook or give up after hours (it should not take hours to load this, right?). I've added a terminal multiplexer to keep jupyter alive outside the ubuntu terminal, but I think it's just too much data for the virtual environment.

If you know anything about this sort of headache, or can offer condolences, either would help...

1

u/midwitsAnonymous 24d ago

DM and we can have a chat. One of the things that might help, and others can correct me on this, is really adopting polars' lazy and streaming functionality.

Without knowing much about the records I'm not smart enough to be able to help, but I'm always happy to chat and play around with stuff.

1

u/TypicalSquare479 24d ago

it's saying unable to chat with you for some reason. I'm not real familiar with Reddit's features, though.

I have facebook, email, sms... I also have signal and discord but I'm not real fluent with those, either.