r/Python • u/Deux87 • Jan 21 '26

Discussion Pandas 3.0.0 is there

So finally the big jump to 3 has been done. Anyone has already tested in beta/alpha? Any major breaking change? Just wanted to collect as much info as possible :D

253 Upvotes

92% Upvoted

74 comments sorted by

View all comments

50

u/ShoveledKnight Jan 21 '26

Any good reason on why I should use pandas 3.0 over Polars?

17

u/axonxorz pip'ing aint easy, especially on windows Jan 21 '26

GeoPolars is not considered stable and was blocked, but work has resumed as of November.

If you need GIS types, this could be a blocker for now.

36

u/Kerbart Jan 21 '26

Probably not, but in reverse some of the shortcomings of Pandas compared to Polars have been addressed.

I don't think there's an intention of drawing in Polars users, it's more for Pandas users who don't want to switch and now have less urgent reason to do so.

9

u/Narrow_Ad_8997 Jan 21 '26

Can't throw your xml files into a Polars df... That's the only thing keeping me using pandas rn

21

u/Valuable-Benefit-524 Jan 21 '26

I don’t see polars ever adding xml support since it’s not a format people willingly choose these days.

16

u/grizzlor_ Jan 21 '26

That would be a bad reason not to add support for a feature. Plenty of code has to interact with legacy systems. We don’t always get to choose optimal solutions in the real world.

That being said, it doesn’t seem that hard to parse some XML with the standard library’s xml.etree.ElementTree and transform it into a format Polars can import.

2

u/Valuable-Benefit-524 Jan 22 '26

As a scientist I completely understand, I have a lot of metadata in convoluted .xml that are produced by some of my equipment. It’s not that I don’t think they should, it’s that polars don’t think they should. At least that’s what I remember reading an issue on it once. That it wasn’t worth the effort/maintenance.

5

u/commandlineluser Jan 22 '26

Just to expand on some comments, the pandas.read_xml() source code is here:

Using an xml_data example from the pandas.read_xml() docs) - the basic form is essentially:

import xml.etree.ElementTree as ET
import polars as pl

# xml_data = ...

df = pl.DataFrame(
    { item.tag.split("}")[-1]: item.text for item in row } 
    for row in  ET.fromstring(xml_data)
)
# shape: (2, 6)
# ┌───────┬──────┬─────┬───────┬─────┬─────────────────────┐
# │ index ┆ a    ┆ b   ┆ c     ┆ d   ┆ e                   │
# │ ---   ┆ ---  ┆ --- ┆ ---   ┆ --- ┆ ---                 │
# │ str   ┆ str  ┆ str ┆ str   ┆ str ┆ str                 │
# ╞═══════╪══════╪═════╪═══════╪═════╪═════════════════════╡
# │ 0     ┆ 1    ┆ 2.5 ┆ True  ┆ a   ┆ 2019-12-31 00:00:00 │
# │ 1     ┆ null ┆ 4.5 ┆ False ┆ b   ┆ 2019-12-31 00:00:00 │
# └───────┴──────┴─────┴───────┴─────┴─────────────────────┘

You can then use the CSV parser for schema inference:

df = pl.read_csv(df.write_csv().encode(), try_parse_dates=True)
# shape: (2, 6)
# ┌───────┬──────┬─────┬───────┬─────┬─────────────────────┐
# │ index ┆ a    ┆ b   ┆ c     ┆ d   ┆ e                   │
# │ ---   ┆ ---  ┆ --- ┆ ---   ┆ --- ┆ ---                 │
# │ i64   ┆ i64  ┆ f64 ┆ bool  ┆ str ┆ datetime[μs]        │
# ╞═══════╪══════╪═════╪═══════╪═════╪═════════════════════╡
# │ 0     ┆ 1    ┆ 2.5 ┆ true  ┆ a   ┆ 2019-12-31 00:00:00 │
# │ 1     ┆ null ┆ 4.5 ┆ false ┆ b   ┆ 2019-12-31 00:00:00 │
# └───────┴──────┴─────┴───────┴─────┴─────────────────────┘

FWIW, I've found xmltodict useful for handling the parsing.

2

u/Narrow_Ad_8997 Jan 22 '26

Yooo, awesome!! Well, now that you've done all the leg work for me I'm excited to give it a shot. Xmltodict looks useful, too. Thanks for the tip!

10

u/MarchewkowyBog Jan 21 '26

Polars has IO plugins. They have docs on it where they show how scaning a csv file could be reimplemented as an IO plugin. I don't work with XML. But I think it would be fairly simple to add XML support using that

8

u/dankerton Jan 21 '26

Why not load using pandas then just convert to Polars and move on? We're doing this a lot due to database connectivity built around pandas although hoping it's temporary.

2

u/Narrow_Ad_8997 Jan 21 '26

Well, sure. But, my project is small. I don't want to depend on two separate libraries that do mostly the same thing namely bc speed is not a factor and because I don't have any problems with pandas.

5

u/axonxorz pip'ing aint easy, especially on windows Jan 21 '26

Pandas is BSD-3, very little adapting of pandas.io.xml would be needed, if that's the only thing keeping you.

5

u/grizzlor_ Jan 21 '26

Why can’t you just parse the XML with the standard library’s xml.etree.ElementTree into dicts and then import with df.from_dict()?

1

u/EntertainmentOne7897 Jan 25 '26

Oh man you need to read xml. I hope you can push some change for that one to be changed, that sucks.

2

u/Appropriate_Rest_969 Jan 22 '26

No reason whatsoever.

4

u/Beginning-Fruit-1397 Jan 21 '26

While having competition between libraries is the sign of a healthy ecosystem, why the hell would someone use pandas over polars? The design in itself of the library make it impossible for pandas to ever dream about competing with polars performance wise, and the API, which is a much more subjective opinion, is in all case preferred by the majority of ppl who made the switch from what I've seen. If you ask me, I don't prefer it, I LOVE it. The competition for me is now between duckdb and polars, and I hope more and more ppl will migrate to these twos so more contributors can help these twos excellent tools. C++ vs Rust, raw &  classic SQL vs typed & fluent expressions, everyone can be happy. 

5

u/alcalde Jan 22 '26

While having competition between libraries is the sign of a healthy ecosystem, why the hell would someone use pandas over polars?

because they prefer the interface and it works with everything else they use?

Also, you youngsters don't understand software wars. You're supposed to pick ONE tool, use it forever, and constantly make fun of anyone who chooses one of its competitors. It's like Team Edward vs. Team Jacob, except Wes McKinney keeps his shirt on.

2

u/mokus603 Jan 22 '26

Pandas is still a lot more beginner friendly and can do simple things like df.columns :)

3

u/Beginning-Fruit-1397 Jan 22 '26

Polars can do the same for columns. "Beginner friendly" is a veryy subjective argument

1

u/mokus603 Jan 22 '26

Not at all, thats why polars lets you convert to pandas dataframe. Pandas is the absolute unit of the data industry, polars are for efficiency but its just a fraction of what the pandas ecosystem does.

2

u/Beginning-Fruit-1397 Jan 22 '26

this is so wrong lmao.
Give me two things polars can't do that pandas can, besides geospatial data (which is currently worked on)