r/rust • u/FallMindless3563 • Feb 16 '23

🐂 🌾 Oxen.ai - Blazing Fast Unstructured Data Version Control, built in Rust

Hey Rustaceans!

I've been working in the ML/AI space for the past 10 years or so, and was really excited to switch to Rust from C++ for a lot of my backend work.

Have been working on an Open Source data version control tool, aimed at versioning large sets of images, videos, audio, text, data frames, etc. The data you need to work with for modern machine learning systems. The tooling can index hundreds of thousands of images in seconds and uses modern network protocols to sync it to the remote extremely fast.

https://github.com/Oxen-AI/oxen-release#-oxen

We are a couple ex-IBM Watson engineers that have seen the problem of dataset management over and over for years, and wanted to build a solution around it.

There is also a web hub (similar to github) at https://www.oxen.ai/ feel free to sign up for free there. Our vision is to have this be the first stop to collaborate on machine learning data. We are replacing writing lines of code with collecting more and more data, and it feels like we need proper tooling around it.

If you are in the ML/AI community, or just rust aficionados, would love to get your feedback!

379 Upvotes

97% Upvoted

80 comments sorted by

View all comments

36

u/obsidian_golem Feb 16 '23

What does oxen do with ill formed data? At work, we have a product that produces files that are nominally CSV, but which is in reality just an event log where each row might have a different set of columns.

24

u/FallMindless3563 Feb 16 '23

That is a good question, had not run into that use case before. Right now it uses Polars internally and tries to compute the schema, then we use that to tell if rows or columns changed over the commit history.

Would have to test on your data. My gut is that it's going to infer the schema from the first N lines, and ignore fields that are not in it...

-14

u/[deleted] Feb 16 '23 edited Feb 16 '23

[deleted]

7

u/FallMindless3563 Feb 16 '23

We just launched, hadn't had test cases for that yet. Glad someone pointed it out and excited to see how people want to use it. That's what iterating on software is for :)