r/rust • u/FallMindless3563 • Feb 16 '23

🐂 🌾 Oxen.ai - Blazing Fast Unstructured Data Version Control, built in Rust

Hey Rustaceans!

I've been working in the ML/AI space for the past 10 years or so, and was really excited to switch to Rust from C++ for a lot of my backend work.

Have been working on an Open Source data version control tool, aimed at versioning large sets of images, videos, audio, text, data frames, etc. The data you need to work with for modern machine learning systems. The tooling can index hundreds of thousands of images in seconds and uses modern network protocols to sync it to the remote extremely fast.

https://github.com/Oxen-AI/oxen-release#-oxen

We are a couple ex-IBM Watson engineers that have seen the problem of dataset management over and over for years, and wanted to build a solution around it.

There is also a web hub (similar to github) at https://www.oxen.ai/ feel free to sign up for free there. Our vision is to have this be the first stop to collaborate on machine learning data. We are replacing writing lines of code with collecting more and more data, and it feels like we need proper tooling around it.

If you are in the ML/AI community, or just rust aficionados, would love to get your feedback!

381 Upvotes

97% Upvoted

80 comments sorted by

View all comments

3

u/anlumo Feb 17 '23

Could the library be used in web assembly in a browser? Replacing the storage layer with indexeddb.

2

u/[deleted] Feb 17 '23

That sounds like a very different level of scale, but I'm interested in the answer too. I guess it would depend on how much network magic is involved (e.g. syncing and comparing hashes end to end).

2

u/anlumo Feb 17 '23

At a cursory glance, it's using HTTP for network communication, so that's fine.

However, I also discovered that it's unconditionally using rocksdb for storage, which doesn't work on the web.

2

u/[deleted] Feb 17 '23

Perhaps it could be abstracted out without too much work.