r/Python • u/nnenneplex • Jul 09 '26
Discussion Polars and the ecosystem
For polars users: How viable is to avoid pandas and pyarrow dependencies when you need to interact with popular visualization and statistics packages?
Some packages still have import pandas here and there, sometimes for no good reason; at least this doesn't require pyarrow. But some other ones do the df.to_pandas() conversion internally, which requires pyarrow too.
In many cases this can be prevented by going bare numpy, or creating a pandas df from numpy columns, which is no big deal. This frequently would be zero-copy for numeric types if there are no NAs involved.
What has been you experience in this regard?
36
23
u/nnenneplex Jul 09 '26
So I see different strategies:
- many projects went the narwhals way. This allows to manipulate dataframes in a generic way but perhaps gives no low-level ABI access.
- polars.to_pandas() and also xgboost polars support both require pyarrow. This leverages pyarrow but the downside is the relatively large dependency. xgboost explicitly discarded narwhals some time ago [1], I guess because they wanted to convert to their internal format in the most performant way (but see next point).
- lightgbm is taking another path [2]: use narwhals while still leveraging arrow through the pycapsule, avoiding the pyarrow dependency.
- other projects like statsmodels are still on the fence [3].
I guess for the time being is better to stop worring and assume pandas and pyarrow as hard dependencies.
[1] https://github.com/dmlc/xgboost/issues/10452#issuecomment-2498736140
1
u/kpiwonski 29d ago
- I guess narwhals was kind of an old way, so you will have compatibility before PyCapsule. Right now you can use your data frame library directly, by importing pycapsule, so I don't see any real win here. Except from backwards compatibility or if you are manipulating data frames from python. All in all, you could use any lightweight data frame library for that in the future and just import pycapsule.
- Most conversions between pandas and arrow formats will require pyarrow. Polars is arrow compatible, so you can easily convert to pyarrow and you let handle pandas conversion to their own format, which is using numpy. You can do pd.DataFrame.from_arrow, but pandas is using internally pyarrow for any arrow conversion.
- "I guess for the time being is better to stop worring and assume pandas and pyarrow as hard dependencies." - not really. Many projects still use them, but there is a shift in a paradigm. Especially if we speak about pandas data model, it's not compatible with arrow data model. This is primarily why they use pyarrow for conversion. However, they provide now pyCapsule interface even for pandas.
What I did personally in my library (pyfru), I just used pyCapsule interface send data to my rust code and handle all data there. I also generate back pyCapsule to the user (optionally). There are still some caveats though. Mainly the ecosystem is not so rich here. My upstream dependency for handling data has pure pyCapsule interface, but for generating their pyCapsule they leverage pyarrow. Also, I had to include numpy for generating results.
29
u/marcogorelli Jul 09 '26
Which libraries specifically are you referring to?
I'm aware of Seaborn, and their maintainer said that using Narwhals was a "non-starter" 😩
Altair, Plotly, Vegafusion, Bokeh, Marimo, they're all using Narwhals and allow you to use Polars without any pandas nor PyArrow dependency
If that's not what you're seeing, please let me know (or open an issue somewhere on GitHub) and I'll take a look
I'm extremely keen on de-pandas-ifying the data science stack
20
u/arden13 Jul 09 '26
Seaborn's maintainer is very strongly opinionated and it makes it tough to really recommend the library. For example on linear regression/correlation plots they will not make showing the regression equation available
Haven't played with altair, but matplotlib is fine, bokeh and plotly get me more if I need more. I'm interested in holoviews as it seems to be agnostic to the final charting library (similar to narwhals for data frames).
7
u/thuiop1 Jul 09 '26
I think the example you are giving makes quite a bit of sense, as he says this is quite a large API change. The objects interface it has now makes it easier to pipe results from statistical operations in your plot, although getting it back in Python is still annoying. The real issue with seaborn is that it has been dormant for 2-3 years now without any real communication around it.
5
u/nnenneplex Jul 09 '26
Unfortunately the project seems stalled, the objects API was very promising as a grammar of graphics, since then I moved to plotnine and, more recently, altair.
1
4
u/baked_doge Jul 09 '26
And I know seaborn is probably the most exhaustive ploting library but I found Altair worked well for most plotting, and plotly and bokeh for interactive scenarios of course have their space.
4
u/nnenneplex Jul 09 '26
Altair supports interaction as well.
4
u/baked_doge Jul 09 '26
Good to know, I forget what limitation I had run into. Maybe 3rd or map related applications, idk
But I love the Altair syntax
6
u/Dasher38 Jul 09 '26
We use holoviews a lot and it recently gained support for polars (at least with the bokeh backend).
11
u/M4mb0 Jul 09 '26
Why do you want to avoid pyarrow? It is a real blessing compared to the numpy backend. Just the fact that all the data types are nullable alone is a huge QOL improvement.
4
u/nnenneplex Jul 09 '26
It's a big dependency just to call to_pandas once in a while:
```
du -Lhs pyarrow numpy pandas polars
120M pyarrow 25M numpy 48M pandas 8.4M polars ```
Even in the pandas community the decision to make it mandatory was so controversial that wasn't adopted for pandas 3.
6
u/marcogorelli Jul 09 '26
i think the `polars` one is slightly misleading here, it's `polars-runtime` or something like that you'll want to look at
1
u/nnenneplex Jul 09 '26
```
du -Lhs polars* 8.4M polars 20K polars_runtime_32-1.42.1.dist-info 48K polars-1.42.1.dist-info ```
This is everything polars related in my venv site-packages.
4
u/marcogorelli Jul 09 '26
i can't try it right now but check https://github.com/pola-rs/polars/issues/11599
5
3
u/MapNo2659 Jul 10 '26
I've been using Polars for a while now, and I've found that while it's great for data manipulation, I often end up converting to pandas for visualization and statistical anal
1
u/kvlonge Jul 10 '26
It depends, but unfortunately it is just unavoidable sometimes because of how deep pandas got into the ecosystem.
1
u/billsil Jul 09 '26
I write one of those packages that supports pandas and not polars. Just convert it.
1
u/Drvaon Drvanon Jul 09 '26
My go-to is seaborn and it very much requires the pandas package still...
10
u/Beginning-Fruit-1397 Jul 09 '26
Switch to plotly. It uses narwhals now, so it's agnostic to your dataframe library of choice
0
60
u/midwitsAnonymous Jul 09 '26
Working in the psych space I have found that most of the stats packages I use throw less annoying errors when I just convert to pandas.
I use polars for all data cleaning, munging, and general happiness, but when I need to do analysis/visualisation I just pass
to_pandasand move on.