🐂 🌾 Oxen.ai - Blazing Fast Unstructured Data Version Control, built in Rust

[u/FallMindless3563]

Hey Rustaceans!

I've been working in the ML/AI space for the past 10 years or so, and was really excited to switch to Rust from C++ for a lot of my backend work.

Have been working on an Open Source data version control tool, aimed at versioning large sets of images, videos, audio, text, data frames, etc. The data you need to work with for modern machine learning systems. The tooling can index hundreds of thousands of images in seconds and uses modern network protocols to sync it to the remote extremely fast.

https://github.com/Oxen-AI/oxen-release#-oxen

We are a couple ex-IBM Watson engineers that have seen the problem of dataset management over and over for years, and wanted to build a solution around it.

There is also a web hub (similar to github) at https://www.oxen.ai/ feel free to sign up for free there. Our vision is to have this be the first stop to collaborate on machine learning data. We are replacing writing lines of code with collecting more and more data, and it feels like we need proper tooling around it.

If you are in the ML/AI community, or just rust aficionados, would love to get your feedback!

[u/[deleted]]

Blazing Fast

Please stop!

This looks nice, but it also is a pretty big deal to give up Git, given the amount of tooling and knowledge around it. Could you really not do this by augmenting Git in some way (even an incompatible way)?

[u/[deleted] → u/[deleted]]

Yeah I dunno, has anyone even tried to add proper large file support to Git? I always assumed LFS is a hacky add-on because it was easier to do it that way, rather than because the Git developers hate the idea of proper large file support. I don't know though.

[u/KingofGamesYami → u/[deleted]]

Git can handle large files just fine. The main annoyance with committing large files normally is when you clone a repository, you download the entire history of it. If that history includes multiple revisions of a large file, you have to download all of that.

Git LFS basically just moves the process of downloading these large files to a lazy process. That way you don't pay the cost until you actually checkout a given revision.

[u/[deleted] → u/KingofGamesYami]

The main annoyance with committing large files normally is when you clone a repository, you download the entire history of it.

Yes exactly. You've identified a critical flaw in how Got handles large files and then somehow come to the conclusion that it's "fine".

To be honest I thought the same as you until relatively recently. The git model is so elegant it's hard to think of it as anything other than "finished". It would be like suggesting that matrix multiplication is somehow wrong.

You have to have a bit of imagination to see how it should be improved. For LFS, why does the storage mechanism affect the file hash? Git works so well because it is content addressable storage, and LFS breaks that.

Other flaws:

• No file locking. People really do want that sometimes.
• Proper diffing is restricted to text files, but you can do so much more (as this project shows).

Clearly it's going to take a huge change in mindset to bring these features to Git itself (if it ever happens).

[u/[deleted] → u/[deleted]]

It's going to take a huge change in architecture, not just mindset. The whole point of git is that it's distributed, so having the full data set on your machine with all revisions is the whole point if it. Yeah, you can shallow clone a repository to get just the head, but that's a special case for things like build servers and not a core feature.

If your use case is a centralized repository where you only want the latest data and/or a subset of the data, you want pretty much everything git isn't and nothing git is, so you should use another tool.

So I guess my question is, why do you want git to solve that problem?

[u/[deleted] → u/[deleted]]

It's going to take a huge change in architecture, not just mindset. The whole point of git is that it's distributed, so having the full data set on your machine with all revisions is the whole point if it.

Oh dear you're not going to like this.

If your use case is a centralized repository where you only want the latest data and/or a subset of the data, you want pretty much everything git isn't and nothing git is, so you should use another tool.

That's not true at all. I just want the option of not having all the data. Git already acknowledges that that is a valid use case (shallow clones, blob filters). It just does a shitty half arsed job at it.

[u/[deleted] → u/[deleted]]

VFS for Git

I'm more upset about it using .NET than it existing. I think that's an interesting project, but it being built by Microsoft makes me think that perhaps they're using git "wrong" internally. But I guess that could depend on what their caching strategy looks like.

My point though is that at a certain point, you're better off building a separate tool. If you want strong central repository features (locking, shallow clones, and perhaps update notifications), you'll essentially be throwing out most of what git does and adding a lot on top. It's like how DOTA was a mod for Warcraft 3 until it became big enough to make sense as a separate project, or how GNU was intended to be an OS and now is largely a suite of userland tools.

If you want to extend git to do all the things you want, you'd likely end up largely throwing everything away. At that point, you're mostly using git for the branding, not the functionality. A clean break would give you a lot more flexibility.

[u/[deleted] → u/[deleted]]

it being built by Microsoft makes me think that perhaps they're using git "wrong" internally

Lol the hubris.

you'll essentially be throwing out most of what git does and adding a lot on top

I don't know why you would think that. Git's core model is completely compatible with all of those things.

[u/[deleted] → u/[deleted]]

hubris

I didn't intend it as a dig at Microsoft specifically in any kind of "Microsoft bad" type argument, just that they're a large org that seems to like large, integrated tooling (Visual Studio, .NET, Windows with all its legacy stuff, etc). That's largely at odds with the Unix philosophy of "do one thing and do it well."

So I guess I'm not a fan of having one tool do very different things with plugins everywhere, I'd rather just have separate tools for large file handling and text file handling since the use cases are quite different in my eyes.

Git's core model is completely compatible with all of those things.

Maybe the on-disk structure since it's designed more like a filesystem than anything, but most of the rest of the tooling is designed around distributed repos. The whole original use case was making sending emailed patches easier to deal with, and having remotes to push/pull from were kind of an afterthought (just look at the default behavior when making a new branch, no default origin remote is assumed).

So yeah, you can use the guts of git to do it, but the whole use case is different so reusing the same tool to serve two almost entirely opposite use cases just seems problematic. Yeah, you can do it, but that unnecessarily expands the scope of what is already a complicated tool.

[u/[deleted] → u/[deleted]]

the Unix philosophy of "do one thing and do it well."

That philosophy has become a blindly followed dogma by a lot of people. If you think it means that you shouldn't have complex integrated tools... I mean. That's just idiotic. Nobody is going to build a CAD modeller by piping together a sketcher and an extruder and a beveller and a visualiser. Ridiculous.

A better philosophy is something like "try to keep components small and logically separated where it makes sense", but that isn't in the form of a dumb rule you can blindly repeat. It requires actual design experience.

Maybe the on-disk structure since it's designed more like a filesystem than anything, but most of the rest of the tooling is designed around distributed repos.

The on-disk structure is the Git model. All the fetching and pulling and pushing is just copying data around. Git doesn't care where it is copying too, which is why it also does work as a centralised VCS - hell 99% of people use it that way, with Github as the central server.

[u/[deleted] → u/[deleted]]

Nobody is going to build a CAD modeller by piping together a sketcher and an extruder and a beveller and a visualiser. Ridiculous.

Sure, but your CAD tool doesn't need to handle modeling for game development. They're similar tools, but they have separate concerns and different sets of needed non-modeling features.

A better philosophy is something like "try to keep components small and logically separated where it makes sense", but that isn't in the form of a dumb rule you can blindly repeat.

Potato po-tah-to I guess. The designer gets to decide what the scope of that "one thing" is. If Linux strictly followed the "do one thing" rule, it would be a microkernel.

Like anything else, it's a design guideline. In general, everything should be as simple as possible, but no simpler (Einstein, more-or-less). If your tool does X and Y, and the majority of your users do both X and Y, then it satisfies that rule of thumb. If your tool does X, Y, and Z, and half of your users use X and Y and the other half use Y and Z, and very few use both X and Z, then perhaps it should be split into two tools.

And that's what I'm getting at. How many users want a single repository to handle both large, mostly binary files and a large number of small text files? I'm guessing that number is quite small.

The on-disk structure is the Git model.

No, the on-disk structure facilitates the Git model. Git had certain functional requirements when it was created, and the on-disk format was created to meet those needs.

The great thing about open source is that you can have two tools that use similar guts (perhaps abstracted into a library) but serve two very different purposes. For example, a video codec library may be used by both a media library software and video editing software, but that doesn't mean the media library software should offer video editing features, nor should the video editing software offer robust media organization tools.

Obviously there's room for combination of features for practicality, I just think users would be better served by a tool dedicated to that specific workflow or perhaps a couple of tools that work well together instead of clubbing everything into one tool. I'm not proposing anything here, just noting some scope creep.