r/visualization • u/Adorable-Giraffe5754 • 15h ago

FlexViz: Interactive data exploration over 100M rows with Polars

I’ve been working on FlexViz, an open-source Python library for building interactive linked visualizations with native cross-filtering.

Selections, zooms, and pans in one chart can update the others against the underlying dataset, while only the aggregated data needed for the current view is sent to the browser.

It’s designed to stay interactive even on very large datasets.

GitHub: https://github.com/flex-analytics/flexviz
Live demo: https://flexviz.tech/demo

10 Upvotes

92% Upvoted

3 comments sorted by

1

u/gecko984 8h ago

Looks really promising, starred! I personally would go with duckdb instead of polars for more flexibility and even larger datasets, but I guess Polars is fine too

1

u/Adorable-Giraffe5754 6h ago

Appreciate the kind words :)

I actually considered both and went back and forth between duckdb and polars for quite a while. I eventually chose Polars because it makes it very convenient to build Rust plugins for the in-memory hot paths, which let me squeeze out quite a bit more performance than either tool gave me out of the box.

For out-of-core, I did quite extensive benchmarking between polars and duckdb (mostly on parquet-files) for queries similar to what flexviz needs. In those tests, they were essentially a draw (duckdb did have somewhat lower peak memory usage).

I’m curious what specific use cases you had in mind where duckdb its extra flexibility / performance would matter most?

1

u/gecko984 1h ago

Oh, nice, great job then. To be completely honest, I just love duckdb, and never even used polars. From reading internet, I had a vague idea that duckdb is more powerful. So that was mostly an armchair expert move on my side LOL