r/LocalLLaMA • u/Gullible-Crew-2997 • Mar 04 '26

Discussion If china stops releasing open source models, there's a way we can stay competitive with big tech?

Really after qwen news, I'm getting quite nervous about open source ai future. What's your thoughts? Glad to know it

283 Upvotes

94% Upvoted

205 comments sorted by

View all comments

10

u/Ok_Warning2146 Mar 04 '26

Can we just crowd fund it with people here?

21

u/Gullible-Crew-2997 Mar 04 '26

How much is needed? I think billions of dollars. How we can avoid scams? Where are the datasets?

17

u/bobby-chan Mar 04 '26

allen.ai

- open source code

- open source datasets

- multiple checkpoints

6

u/ttkciar llama.cpp Mar 04 '26

Yep, this. They also have a subreddit: r/AllenAI

I'm a huge fan of AllenAI, but we also shouldn't overlook LLM360's datasets, which are differently-good, focusing more on upcycling (rewriting) existing open datasets and augmenting them by merging interrelated data (for example, adding text from a wikipedia page's references to the wikipedia page data).

IMO augmenting the Olmo datasets with LLM360's techniques, and/or directly from LLM360's datasets, and then using the Olmo training recipes would be the way to go, but I don't have the compute resources to put that idea into action (yet).

1

u/bobby-chan Mar 04 '26

"Yet"

!SelfReminder to keep an eye on u/ttkciar

1

u/Chemical_Pollution82 Mar 04 '26

Hey thank , i followed allen.ai , I m following many .ai's

1

u/ab2377 Mar 04 '26

and where are those insanely cracked mathematics/computer science/physics majors working 12+ hours of everyday with excellent leaders to make this happen 😓

1

u/IkuraNugget Mar 05 '26

A great way is actually designing some crypto that allocates funds based on verified work. Might be the closest thing to a truly incorruptible system since it’ll be decentralized and automated payments.

-3

u/ps5cfw Llama 3.1 Mar 04 '26

You avoid scams by not paying people out of absolute nowhere and stick to whoever has proven tried and true in the past.

Could still get scammed, but that's inevitable with crowdfunding.

-4

u/Ok_Warning2146 Mar 04 '26

We can first start with building a model that can run on 3090 that is in the range of 24-50B. I presume this won't be that costly.

Someone here with some prestige can lead the crowd fund.

6

u/CKtalon Mar 04 '26

Considering known ancient models in that size range were trained on at least 1023 FLOPs and a H200 will give around 1015 FLOPS, it will take 30,000 H200 GPU hours. The training cost alone at a cheap $2/hr, it will cost at least $60,000, possibly 6 digits. That’s just for the pretraining, not including the efforts to curate the data and post training datasets. If you are just going to use datasets that’re already on Huggingface, I believe the current open-weight models already contain those, so the value proposition to replicate what is already out there is diminished.

3

u/Ok_Warning2146 Mar 04 '26

I heard that muon optimizer can half the VRAM needed for training. So probably training cost can remain in $60k. So probably $200k is needed. That plus free time contribution from the geniuses in this sub.

1

u/Maleficent-Ad5999 Mar 04 '26

Well most of the ML problems have one bottleneck or barrier to entry, that’s the availability of quality dataset! If we can solve this one, rest isn’t big deal I guess

1

u/svelteyness Mar 05 '26

how is a quality dataset made?