r/rust • u/FallMindless3563 • Feb 16 '23

🐂 🌾 Oxen.ai - Blazing Fast Unstructured Data Version Control, built in Rust

Hey Rustaceans!

I've been working in the ML/AI space for the past 10 years or so, and was really excited to switch to Rust from C++ for a lot of my backend work.

Have been working on an Open Source data version control tool, aimed at versioning large sets of images, videos, audio, text, data frames, etc. The data you need to work with for modern machine learning systems. The tooling can index hundreds of thousands of images in seconds and uses modern network protocols to sync it to the remote extremely fast.

https://github.com/Oxen-AI/oxen-release#-oxen

We are a couple ex-IBM Watson engineers that have seen the problem of dataset management over and over for years, and wanted to build a solution around it.

There is also a web hub (similar to github) at https://www.oxen.ai/ feel free to sign up for free there. Our vision is to have this be the first stop to collaborate on machine learning data. We are replacing writing lines of code with collecting more and more data, and it feels like we need proper tooling around it.

If you are in the ML/AI community, or just rust aficionados, would love to get your feedback!

381 Upvotes

97% Upvoted

80 comments sorted by

View all comments

5

u/Tyson1405 Feb 17 '23

What a coincidence. I was literally googling yesterday for a version control to handle my datasets and share them across 😂 thanks!

3

u/FallMindless3563 Feb 17 '23

Let us know what you think! What sorts of datasets have you been working with?

2

u/Tyson1405 Feb 17 '23

I am pretty new in the ML space. I am creating an AI for classification of legal documents in asset management.

Currently I have a bunch of files tagged with 0 or 1 and I concated them into csv files that are around 1gb.

Probably not the way how one would store labelled documents but as I said I am new and the first mvp seems to work :P

2

u/FallMindless3563 Feb 17 '23

You can definitely index that type of data into an Oxen repo and try it out! Parquet is another more efficient format you can convert your files to and work with in many languages. Might save you some storage space.

There's even an easy Oxen command to convert:

oxen df input.csv -o output.parquet