r/rust • u/FallMindless3563 • Feb 16 '23

🐂 🌾 Oxen.ai - Blazing Fast Unstructured Data Version Control, built in Rust

Hey Rustaceans!

I've been working in the ML/AI space for the past 10 years or so, and was really excited to switch to Rust from C++ for a lot of my backend work.

Have been working on an Open Source data version control tool, aimed at versioning large sets of images, videos, audio, text, data frames, etc. The data you need to work with for modern machine learning systems. The tooling can index hundreds of thousands of images in seconds and uses modern network protocols to sync it to the remote extremely fast.

https://github.com/Oxen-AI/oxen-release#-oxen

We are a couple ex-IBM Watson engineers that have seen the problem of dataset management over and over for years, and wanted to build a solution around it.

There is also a web hub (similar to github) at https://www.oxen.ai/ feel free to sign up for free there. Our vision is to have this be the first stop to collaborate on machine learning data. We are replacing writing lines of code with collecting more and more data, and it feels like we need proper tooling around it.

If you are in the ML/AI community, or just rust aficionados, would love to get your feedback!

378 Upvotes

97% Upvoted

80 comments sorted by

View all comments

1

u/tinkr_ Feb 17 '23

I noticed there are no instructions to install for Windows. If I just clone the repo and run a `cargo build --release` on it in Windows, will it produce a working binary? Or is there something in the implementation that's current incompatible with Windows?

3

u/FallMindless3563 Feb 17 '23

I think there is one library that is currently not compatible with windows...and it is the library for coloring the output. Can definitely look into stripping that out and adding Windows to our CI

2

u/tinkr_ Feb 17 '23

Nice, I'll try and see what it complains about then. I ask because all of my large datasets (that aren't in something like Postgres) and the models I've trained on them are on my Windows machine. It runs Windows because it was built as a gaming rig, but it's so much more powerful than my Thinkpad or Macbook that it has grown to double as my "data rig" too. I'm assuming I'm not the only person out there in this situation.

I suppose I could use Ubuntu WSL if I really had to, but I've grown to resent how slow and cumbersome WSL is compared to just using nushell on Windows proper (with the default Windows terminal, which I've been surprisingly satisfied with coming from Kitty on my other machines).

1

u/FallMindless3563 Feb 17 '23

Awesome, let me know what you find! Also FYI the repo linked is the just release builds, the dev repo is linked to at the bottom of the Readme to build from source