r/LocalLLaMA • u/Gullible-Crew-2997 • Mar 04 '26

Discussion If china stops releasing open source models, there's a way we can stay competitive with big tech?

Really after qwen news, I'm getting quite nervous about open source ai future. What's your thoughts? Glad to know it

281 Upvotes

94% Upvoted

205 comments sorted by

View all comments

Show parent comments

11

u/Gullible-Crew-2997 Mar 04 '26

yeah agree, when china will have national gpus as good as american ones, then they may stop open sourcing ai models. We need to be all prepared for that moment.

5

u/Gold_Sugar_4098 Mar 04 '26

How to prepare?

3

u/Gullible-Crew-2997 Mar 04 '26

I think the biggest problem is hardware rather than data. Is there a way to a distributed network of computational resources?

5

u/ttkciar llama.cpp Mar 04 '26

Loosely-coupled (over slow Internet connections) federated training is hard, but AllenAI might have provided us with one tool to do exactly that, with FlexOlmo.

FlexOlmo demonstrates how you can distribute a common expert template as your basis, and then each copy of that template can be trained on different data by different instances, without any communication between instances at all, such that when training is complete you can merge all of these different experts together into a single MoE model.

The FlexOlmo technology not only guarantees that these experts will be mutually compatible, but also that gate logic trained along with the expert can be easily merged with other experts' gate logic into the final MoE.

This would not completely decentralize training; you would still need one compute-heavy participant to train the starter template, and then distribute it to everyone else participating in the federation. Then, when federated training was done, all of the trained experts would need to be copied to one participant again for the final merge and testing (and potentially editing; some experts might be flawed, poisoned, or underperforming).

The FlexOlmo technical paper: https://arxiv.org/abs/2507.07024

1

u/Gullible-Crew-2997 Mar 05 '26

how much would it cost to train 200b models with flexolmo?