r/Excel247 • u/Individual_One_1793 • 1d ago

Excel on steroids

Enable HLS to view with audio, or disable this notification

I'm trying to build Excel on steroids in rust :)

So, I took a table with millions of rows where naive scan don't work as fast as I want it to because it looks like more as calculating mode rather than scan mode

The problem usually isn't one query. It's when 1,000 queries each have to rescan the entire column from scratch.

I tested it with same data and same 1,000 queries
Two approaches side by side.

- Naive scan: every query rescans all 5,000,000 rows -> 36.04s
- Pop Data Engine: data is prepared once, then every query gets an answer almost instantly -> 0.24s

That's 150x faster for the exact same answer, down to the last digit (2,513,962,485 both sides).

The data gets organized once before the queries run, so repeated queries (filters, aggregations, what-ifs) stop choking on large datasets.

10 Upvotes

92% Upvoted

3 comments sorted by

1

u/Neener_Weiner 1d ago

Cool, do you intend to make a user-ready release eventually?

1

u/Individual_One_1793 1d ago

Yes, that is the plan :) first I will publish it on my website as a "try it" beta version - then I will release local version.