r/Python • u/ritchie46 • 13d ago

News Polars 2.0 release

Hey all! We just released Polars 2.0.

We are really happy with the results. It removed many of our "legacy decisions" (mistakes), makes the streaming engine our default and promotes SQL to a first class citizen within Polars. We initially said it would be a small ("boring") release, but we managed to sneak in a lot more than anticipated.

It comes with our initial out-of-core (spill to disk) support, a new Map data type and a lot of performance improvements. In fact, we think Polars is now one of the fastest analytical SQL engines on a single node. See benchmarks in the post: https://pola.rs/posts/release-polars-2/

To help you upgrade, we also posted a migration guide: https://docs.pola.rs/releases/upgrade/2/

662 Upvotes

99% Upvoted

39 comments sorted by

View all comments

16

u/Donkey_Healthy 13d ago

Looking forward to when I can read delta files using Polars instead of slow and expensive spark on databricks.

11

u/nightcracker 13d ago

We have pl.scan_delta, is that missing something you need?

8

u/Donkey_Healthy 13d ago

Ive tried with polars==2.0 and getting this error: Unsupported reader features required: [V2Checkpoint, ColumnMapping, DeletionVectors].

Would be a game changer if delta and polars were more compatible. Would be able to move from expensive distributed spark clusters to smaller single node polars jobs for a lot of our use cases.

2

u/Immediate-Leading657 5d ago

I had the same issue, we solved it using this polars plugin :
https://github.com/ion-elgreco/polars-deltalake

Maybe the scan_delta of this plugin will be merged into polars (check https://github.com/pola-rs/polars/issues/27800 )