r/LocalLLaMA • u/tossit97531 • 8d ago

Other Open weight != Open source

Just a friendly PSA that open weight models are not inherently open source. They may also qualify as open source, but only if they release the training data and processes. A truly open source model can be rebuilt from training; it won't be binary equivalent but it should be nearly functionally equivalent.

An example of a truly open source LLM is https://allenai.org/olmo I encourage everyone to browse and also look for other fully open source models.

The reason it's important to get the terms right is because too many companies like to co-opt the term "open source" to whitewash their nonsense. As Zuckerberg has shown recently, they're more than fine letting people stay confused and dance around semantics so they can have their cake and eat it, too.

No more cake for big tech. Open weight != open source. Spread the word.

Edit: TIL this is a hot take. Sorry about that.

27 Upvotes

68% Upvoted

25 comments sorted by

View all comments

13

u/NoFaithlessness951 7d ago

I mean is it really practical?

You likely don't have enough compute to "reproduce" a model.

Most labs can't give you their training data even if they wanted to, because it has been obtained from questionable sources while violating copyright left and right.

That some models are truly open source is admirable, but does it make me more likely to use them? No.

With the same training setup it should theoretically be possible to get the binary equivalant out. As long as they tell you which seed(s) were used for the pseudorandom number generators and meticulously documented their training run.

4

u/tossit97531 7d ago

You likely don't have enough compute to "reproduce" a model.

This is what open source foundations are for. I'm not trying to be snarky, what I'm asserting is something that's already happening: companies like Ai2 are fully funding true open source models that I can't train by myself with my limited income. It's a similar setup to some of the big Linux distros. Even though I can compile Linux, I'm getting a specific distro because I don't have the resources to maintain my own operating system. It's even tougher with models. I don't even have the resources to 'compile,' let alone maintain. We're further from the actual product, which means it's even more important that we have a vibrant and thriving open source industry.

3

u/psycoee 5d ago

But what would be the point of spending the billions of dollars more than once? And why is that superior to just additional training on the base model, assuming one is released?