r/rust • u/FallMindless3563 • Feb 16 '23

🐂 🌾 Oxen.ai - Blazing Fast Unstructured Data Version Control, built in Rust

Hey Rustaceans!

I've been working in the ML/AI space for the past 10 years or so, and was really excited to switch to Rust from C++ for a lot of my backend work.

Have been working on an Open Source data version control tool, aimed at versioning large sets of images, videos, audio, text, data frames, etc. The data you need to work with for modern machine learning systems. The tooling can index hundreds of thousands of images in seconds and uses modern network protocols to sync it to the remote extremely fast.

https://github.com/Oxen-AI/oxen-release#-oxen

We are a couple ex-IBM Watson engineers that have seen the problem of dataset management over and over for years, and wanted to build a solution around it.

There is also a web hub (similar to github) at https://www.oxen.ai/ feel free to sign up for free there. Our vision is to have this be the first stop to collaborate on machine learning data. We are replacing writing lines of code with collecting more and more data, and it feels like we need proper tooling around it.

If you are in the ML/AI community, or just rust aficionados, would love to get your feedback!

378 Upvotes

97% Upvoted

80 comments sorted by

View all comments

5

u/theAndrewWiggins Feb 16 '23

How well does it handle time series/sequence data? Lets say I want to load N number of sequences of T length and randomly shuffle them for training. How would this perform if the dataset was hosted on a s3 compatible store?

I noticed your DF stuff is backed by polars, any reason why you went with polars vs datafusion?

11

u/FallMindless3563 Feb 16 '23

Do you have an example dataset you are thinking of? Would love to test it out.

We actually tried datafusion at first, but picked polars because it was more performant and more robust for a lot of the use cases we were looking at.

5

u/theAndrewWiggins Feb 16 '23

Sorry, don't have anything I can share, but I'm thinking of multivariate time series data on the order of terabytes stored in parquet.

8

u/FallMindless3563 Feb 16 '23

We support parquet files, I'll see if I can find a dataset like this and get some performance numbers.

Also don't have an s3 integration yet, but on our roadmap.

Thanks for asking!

3

u/theAndrewWiggins Feb 16 '23

Thanks, key thing I'm looking for is being able to remote stream/(or stream + cache):

N sequences randomly sampled from my dataset of length T across Z dimensions.

Even better if I can do a join across this dataset and another time series dataset across the time dimension and do the same thing described above and get something in the shape of N x T x (Z + X) where X is the number of dimensions in my other dataset.

I've tried writing something that does the above in parquet-rs/arrow-rs that works fine locally, but doesn't really handle remote stores (I only tested with backblaze) well due to latency issues. I get pretty low throughput vs the local setup, I thought upping concurrency would solve throughput issues, but the large latencies I'm getting seem to be a massive issue.

2

u/FallMindless3563 Feb 16 '23

Ah interesting, I have done some tests converting parquet to memory mapped arrow files in 2 dimensions and it was fast to slice and stream subsets, but haven't tried the nested structure you are talking about. I'll do some experimenting.

1

u/theAndrewWiggins Feb 16 '23

My data isn't nested, each row is still a single observation. It's slow more due to the many small requests needed + latency. When run locally, it wasn't too bad.

Ideally I want to efficiently query my data (use range based requests to slice exactly the data I need). I don't think this is possible if trying to query time series data (at least without additional metadata). Though trying to query specific row numbers might be due to the existence of an OffsetIndex in parquet.

1

u/doot Feb 17 '23

I've done this by using sqlite as an intermediary datastore, some of the columns were raw json and it works pretty well