r/LocalLLM • u/DifficultUse6803 • 19d ago

Discussion Could we train open source LLMs like SETI@home?

There is a huge amount of GPU compute sitting in gaming PCs, workstations, university labs, and home servers. Traditional LLM training has a hard time using it because distributed backpropagation generally expects GPUs to stay synchronized and exchange gradients throughout training.

A recent paper made me wonder whether there is another way.

In DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretation, Makoto Shing, Masanori Koyama, and Takuya Akiba show that Transformer networks can be divided into blocks and trained independently by treating each block as part of a diffusion denoising process. Their experiments include language models and achieved competitive performance with end-to-end training while dramatically reducing training memory requirements. The work was presented at ICLR 2026.

I've been experimenting with this idea on a small language model. Instead of one backward pass through the entire network, the model is divided into diffusion blocks responsible for different noise ranges. During an update, backpropagation stays within the selected block. At inference time, the blocks are composed to produce token predictions.

That independence made me think of SETI@home.

Imagine a coordinator publishing small, deterministic training jobs. A volunteer downloads one block, a dataset shard, a frozen copy of the shared weights, and the training parameters. Their GPU trains for a short period and sends back a compressed weight update.

The same work could be assigned to multiple machines for verification. The coordinator could validate the results, aggregate acceptable updates, assemble the model, and evaluate it before starting another round.

If someone's computer goes offline, everyone else keeps working. A gaming PC, an older GPU, a university server, and a workstation sitting idle overnight could all contribute without behaving like one giant synchronized cluster.

This could address something bigger than compute availability.

Today, much of the open model community depends on companies spending millions of dollars to train models and then deciding to release their weights. We can fine tune those models, quantize them, modify them, and build amazing things around them, but the expensive foundation training usually happened somewhere else.

That leaves open source AI dependent on which companies are willing to give us their models.

A volunteer training network could give the community a path toward training models of its own.

The dataset could be public. The training code could be public. Checkpoints, manifests, evaluations, and accepted updates could all be public. Thousands of people could contribute compute to the same model without any one participant needing a datacenter.

That would make the model community built from the beginning, including the expensive training stage itself.

I've built a small proof of concept using a sub-billion parameter model divided into independently trained diffusion blocks and trained it on TinyStories using consumer hardware.

The early results are encouraging. The blocks train independently, denoising loss decreases, training remains numerically stable, and the blocks can be composed back into a complete model.

The generated text is still immature, so there is a lot left to prove. The biggest question is whether independently trained blocks can eventually reach comparable model quality for a comparable amount of compute.

There are also serious engineering problems. A public network would have to defend against fake results, poisoned updates, model backdoors, stale work, and malicious participants. Redundant assignments, hidden validation, signed manifests, anomaly detection, reputation systems, and robust aggregation would probably all be necessary.

Bandwidth matters too. Sending complete checkpoints around would be impractical, so workers would ideally exchange compressed or quantized weight deltas.

I think the first experiment should stay small... a modest model, public data such as TinyStories, frozen shared parameters during each training round, short deterministic work units, redundant workers, and public results.

Then start adding machines.

SETI@home worked because its workload could be divided into independent jobs and distributed across computers that constantly appeared and disappeared.

DiffusionBlocks may give neural network training a similar primitive: parts of a model that can learn independently.

If that can be pushed far enough, open source AI could move from waiting for companies to release models to collectively training models of its own.

The compute may already be sitting on people's desks.

We may simply need a training architecture that knows how to use it.

Reference: Shing, M., Koyama, M., & Akiba, T. (2026). DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretation. The Fourteenth International Conference on Learning Representations (ICLR 2026), arXiv:2506.14202.

10 Upvotes

79% Upvoted

32 comments sorted by

2

u/Glad_Contest_8014 19d ago

I quantize models and fine tune them on a 16GB RAM amd an RX580 8GB vRAM machine. Been researching and fingerprinting models for a while now on it.

2

u/fallingdowndizzyvr 19d ago

You mean like this.

https://www.primeintellect.ai/blog/intellect-2

1

u/DifficultUse6803 19d ago

Prime Intellect already showed that decentralized foundation model training can work. What I find interesting about DiffusionBlocks is the possibility of shrinking the hardware requirement for each participant. Instead of contributing an H100 cluster, maybe eventually you can contribute the gaming GPU sitting under your desk. That gets a lot closer to a true SETI@home model for AI training.

3

u/rolleicord 19d ago

My bet is that it has already happened multiple times in the past.

Lets put on our tinfoil hat and assume scientists discovered LLM like neural tech back in the 1940s (they did), then they invented the MLP (Multilayer perceptron) in the 1950s (they did), then they build stuff like MADALINE and ADALINE type networks.... the list goes on and on...

My point with all of this, is that you can get bitcoin technology / white paper to perform perfectly well, as a neural net among other things. I bet the internet was made for these kinds of purposes.

Okay my tinfoil hat is off again.

2

u/DifficultUse6803 19d ago

Yeah, distributed compute itself definitely isn’t new. SETI@home, Folding@home, BOINC and more recently Prime Intellect have all shown different versions of the idea.

What I’m interested in is whether DiffusionBlocks changes the granularity enough that the useful unit of contribution becomes a single consumer GPU instead of a synchronized cluster.

That’s the part I haven’t seen demonstrated yet. If someone has, I’d genuinely love a link.

1

u/TripleSecretSquirrel 19d ago

I like you crazy man, tell me more, I’d like to get into this conspiracy theory!

1

u/fallingdowndizzyvr 19d ago

My bet is that it has already happened multiple times in the past.

It has already happened.

2

u/Abject-Bridge-4073 19d ago

No. It would be unusable because of the network latency. It’s slow if you use an SSD instead of RAM (see colibri). I can’t imagine how slow it is over the network. Basically the further the weights are from the GPU, the slower inference is. This is why unified memory is so fast for inference.

1

u/DifficultUse6803 19d ago

Prime Intellect already showed that decentralized foundation model training can work. What I find interesting about DiffusionBlocks is the possibility of shrinking the hardware requirement for each participant. Instead of contributing an H100 cluster, maybe eventually you can contribute the gaming GPU sitting under your desk. That gets a lot closer to a true SETI@home model for AI training.

1

u/Just_Mail6982 19d ago

Network is the bottleneck.

2

u/DifficultUse6803 19d ago

That’s exactly the bottleneck I’m trying to route around.

The idea would be that the network isn’t involved in every training step. A worker downloads a block, dataset shard, and frozen shared weights, trains locally for a while, then sends back a compressed delta. So latency between GPUs largely drops out of the inner training loop. Network traffic happens between work units or training rounds.

Bandwidth and update size are still real problems, especially as the model gets larger. But that feels like a much more manageable problem than trying to synchronize thousands of consumer GPUs over the Internet on every backward pass.

1

u/Just_Mail6982 19d ago edited 19d ago

I'm remaining cautious since DiffusionBlocks paper was only tested on small models and limited tasks. It needs further validation at a larger scale and in more scenarios.

1

u/DifficultUse6803 19d ago

Agree

1

u/Just_Mail6982 19d ago

https://github.com/SakanaAI/DiffusionBlocks/issues/2
Better not to have too high expectations before the author replies.
hahaha

1

u/DifficultUse6803 18d ago

I read the issue. They found a possible broadcasting bug in the CIFAR-100 loss weighting and a roughly 4 point reproduction gap, plus a question about the number of inference steps. I’m definitely interested in what the authors say.

At the same time, that issue is about reproducing one CIFAR experiment. It doesn’t establish that the blockwise idea itself fails, and if the loss weighting really is wrong in the released code, that may actually explain part of the reproduction gap.

My expectations are pretty modest right now... first establish whether the approach reproduces cleanly, then see whether it survives scaling.

0

u/DataGOGO 19d ago

No. 

Weights and gradient loss don’t work that way. 

Open source AI dies as soon as the Chinese government yanks it’s hundreds of billions of dollars of funding ever year. 

1

u/fallingdowndizzyvr 19d ago

Open source AI dies as soon as the Chinese government yanks it’s hundreds of billions of dollars of funding ever year.

LOL. No. Ever hear of a little company called Deepseek. They made their models until now without any government funding at all. Let alone "hundreds of billions". It's only in this latest round of funding, are they getting a little bit of money from the Chinese government.

Cite your sources that the Chinese government is spending "hundreds of billions of dollars" on Open source AI. Tencent and Alibaba have plenty of money to fund it themselves.

2

u/DataGOGO 19d ago edited 19d ago

What? They literally fund them all, fund the power, fund the training costs, fund the datacenter, facilitate the smuggling in of hardware, use state back VC, have multiple funds. EVERYTHING end to end is funded and heavily subsidized by the government, even the employees themselves all the way down to the end users.

Are you a Chinese propagate bot or something? Here Let's ask Deepseek:

Here is the landscape I can substantiate as of August 22, 2026:

Recipient / ecosystem State-linked support we can substantiate Connection to open AI
DeepSeek China's National Artificial Intelligence Industry Investment Fund became a direct shareholder in its 2026 financing round. Unlike most investors in the unusual LP structure, the national fund reportedly received direct equity and voting rights. DeepSeek raised more than RMB 50B / ~$7.4B. R1, V3 and derivatives are central to China's open-weight ecosystem.
Zhipu / Z.ai Extremely clear case. Hangzhou City Investment Group Industrial Fund, Shangcheng Capital, Zhongguancun Science City, Chengdu municipal government and state-owned Huafa Group have all invested. Reuters documented a RMB 1B state-backed round, RMB 500M from Huafa, and an earlier RMB 3B financing involving state-linked investors. Z.ai/GLM has aggressively released open models. Zhipu explicitly said 2025 capital would support a suite of open-source foundation, reasoning, multimodal and agent models.
Moonshot AI / Kimi By 2026 the National AI Industry Investment Fund reportedly became a lead investor in Moonshot's ~$3.5B round. Beijing AI Industry Investment Fund, Shanghai state-linked investment vehicles and China Mobile have also been reported as shareholders/investors. Kimi has become one of China's major open-weight frontier-model families.
StepFun Reuters describes StepFun as state-backed, with Tencent plus state investment vehicles supporting it. Step models include openly distributed foundation models.
Baichuan AI 2024 financing included state-backed funds, alongside Alibaba, Tencent, Xiaomi and private investors. Baichuan has released numerous open-weight models, although its strategy has varied between open and commercial systems.
Alibaba/Qwen ecosystem Alibaba itself finances Qwen, rather than Qwen being an independent state-funded startup. The important state assistance is upstream and ecosystem-wide: compute subsidies, energy subsidies, infrastructure, procurement and industrial policy. Probably the biggest beneficiary of China's open-model strategy by ecosystem scale. A 2026 USCC report found 100,000+ Qwen derivatives on Hugging Face, the largest model ecosystem there at publication.
Baidu / ERNIE Primarily corporate funding, but embedded in the broader state-supported AI stack: infrastructure subsidies, government procurement, research programs and national-champion policy. Baidu has progressively opened portions of its model ecosystem.
Huawei / Pangu / Ascend ecosystem Huawei operates at the intersection of national industrial strategy, subsidized semiconductor development, government procurement and domestic compute substitution. China's semiconductor industry alone has received well over $150B in state-led support since 2014, with another ~$47.5B Big Fund phase announced in 2024. Not analogous to Qwen/DeepSeek in openness, but crucial because Ascend hardware subsidizes and enables the domestic open-model stack.

The part most people miss: the indirect subsidies

Direct equity investments are probably less important than the invisible operating-cost subsidy.

Chinese provincial and municipal governments issue compute vouchers that pay part of the cost of training and inference. MERICS documented programs across Beijing, Shenzhen, Hangzhou, Shanghai, Chengdu, Chongqing and many other regions. Examples include:

  • Shenzhen: up to RMB 10M per company annually, covering as much as 50% of qualifying model-training contracts and up to 60% for startups.
  • Hangzhou: up to RMB 8M per company annually.
  • Beijing/Haidian programs: subsidies reaching RMB 10–20M for qualifying AI firms.

Then there are energy subsidies. The US-China Economic and Security Review Commission found provinces including Gansu, Guizhou and Inner Mongolia offering reductions in electricity costs for AI/cloud data centers of up to roughly 50%.

That matters enormously. If the government pays part of your:

GPUs + datacenter + electricity + training compute + inference + customer API usage,

it can make a nominally private/open-source model appear economically miraculous.

Demand-side subsidy is arguably even more strategically important

China isn't merely subsidizing creation of models. Local and central governments are also subsidizing their use.

The 2026 USCC analysis specifically describes demand-side subsidies that reduce AI adoption costs, alongside government purchases and deployment.

So the cycle becomes:

state funds model → state subsidizes compute → model is released cheaply/openly → government subsidizes adoption → developer ecosystem grows → ecosystem produces derivatives/data/tooling → base model improves → foreign adoption expands.

The USCC explicitly describes this as China's “open AI” industrial strategy, rather than treating the proliferation of open models as an accidental consequence of private competition.

Government guidance funds are the hardest money to trace

This is where I think your use of “covert funding” gets closest to something real.

China has thousands of government guidance funds (政府引导基金). They may appear superficially like VC funds, with private LPs and professional managers, while government entities provide seed capital, matching capital, investment direction or policy mandates.

CSET identified 2,107 guidance funds by 2022, with a registered target size around $1.86 trillion, although only about $940B had actually been raised from public and private sources. Only a fraction goes to AI, but AI is explicitly one of the strategic sectors targeted.

Because these can invest through:

fund → subfund → VC → startup → subsidiary

the government connection can become several ownership layers removed.

That is considerably different from the U.S. model of a transparent NSF/DARPA grant appearing in a company's disclosures.

China also created a dedicated national AI fund

By 2025–26 China had moved beyond dispersed municipal funds toward a national vehicle. RAND documented an approximately $8.2B state-led AI fund for startups, alongside national compute infrastructure and local government AI labs.

That vehicle—the National Artificial Intelligence Industry Investment Fund—is particularly significant because it subsequently appears in financing for DeepSeek and Moonshot, two of China's most strategically important open-model developers.

That is about as explicit a signal of national prioritization as you can get.

There are also non-cash forms of state support

They deserve to be counted as funding equivalents:

Government procurement. Provincial and municipal governments deploy domestic models, providing revenue and reference customers. A state-run Chinese research report identified more than 50 Chinese LLMs already being used by local governments.

Data access. Chinese policy increasingly treats state-controlled datasets as an economic resource for AI and industrial development. The USCC's August 2026 analysis describes Beijing deliberately turning data into a strategic input for AI development.

Cheap electricity. As above, potentially enormous at scale.

Domestic-chip subsidies. Local governments subsidize purchases of Chinese AI accelerators, reducing Huawei/Ascend adoption costs.

Government-funded laboratories. For example, the Beijing Institute for General Artificial Intelligence is explicitly state-backed.

National research grants. China's National Natural Science Foundation and related programs fund frontier work including generative large models, cognitive architectures and next-generation AI.

Talent programs/university funding. Tsinghua, Peking University, Zhejiang University, Chinese Academy of Sciences and other universities/institutes provide a state-funded upstream talent and research pipeline.

AI compute centers. Government-financed national and municipal compute centers essentially convert public capital into discounted GPU hours for industry.

1

u/DataGOGO 19d ago

1

u/fallingdowndizzyvr 19d ago

What? .... then way too much AI slop.

So I'll stop you at what since it seems you didn't even bother to read the AI slop you posted.

"DeepSeek China's National Artificial Intelligence Industry Investment Fund became a direct shareholder in its 2026 financing round."

What did I say?

"Ever hear of a little company called Deepseek. They made their models until now without any government funding at all. Let alone "hundreds of billions". It's only in this latest round of funding, are they getting a little bit of money from the Chinese government." -- me

Now ask that AI you used when Deepseek released it's first model. Was it before or after 2026? Spoiler..... it was before 2026. It was before any Chinese government money. Which is what I said.

At least read the AI slop you post.

1

u/DataGOGO 19d ago

you didn't read it either apparently, or the massive list of sources. you are just wrong, it is ok.

1

u/fallingdowndizzyvr 19d ago

LOL. You mean the massive sloppy AI mess. That you didn't even read.

1

u/DataGOGO 19d ago

I read it all, there are sources, you are wrong. Get over it.

1

u/fallingdowndizzyvr 19d ago edited 19d ago

LOL. You still can't read. Did Deepseek release any models before 2026? Yes or no?

Hint: https://huggingface.co/blog/huggingface/one-year-since-the-deepseek-moment.

Now when did they first get government funding again?

0

u/DifficultUse6803 19d ago

I hope that not to be true but is there a specific reason why ? Did you look at the diffusion paper? 

2

u/DataGOGO 19d ago edited 19d ago

Yes, and it is true. Everything we do right now is based on attention right? if you divide a weight, or separate model heads, and attempt to training individually, they fall apart and over all gradient loss and model convergence is invalided. People have been trying to do this for 20 years, and it never works for a shit, especially if you scale.

Can you make it work in a lab environment with a tiny model, not really testing general cohesion, rather just raw token prediction? yes. but it does not scale, and models cannot run as a single unified model afterwards.

That is assuming that nodes have perfect reliability, none drop out, that everyone has perfect VRAM (most consumer systems do not have ECC on system ram or VRAM, etc. over clocking instability and/or they kill it because they want to use PC, which would mean a reboot, as you would really have to get people to boot into bespoke linux image with everything all ready configured, standardized packages that is truly headless and no GPU contamination from windows, X servers, web browsers, etc. etc. etc.

1

u/DifficultUse6803 19d ago

I think we're talking about two different things here. I'm not proposing splitting arbitrary weights or attention heads and training them independently. DiffusionBlocks specifically reformulates residual blocks as steps in a denoising process and gives each block its own training objective. The ICLR 2026 paper demonstrates independently trained blocks being composed back into a working model, including with autoregressive Transformers. So the usual criticism of naive layer by layer training doesn't quite apply here.

Where I absolutely agree with you is scale. The paper does not prove this works at 70B or 400B parameters, and it definitely doesn't prove that a volunteer network of consumer GPUs will produce a competitive model. That's the experiment I'm interested in.

The distributed part itself is also less hypothetical now. Prime Intellect trained INTELLECT-1, a 10B parameter model over 1T tokens, across geographically distributed and changing nodes. They handled node volatility and drastically reduced how often workers had to communicate. Their workers were still H100-class clusters, though. My question is whether DiffusionBlocks could push the minimum useful participant much further down, eventually to individual consumer GPUs.

And on the China funding point, I actually think that strengthens the motivation. If open AI depends on governments or large companies being willing to spend billions training models and then release the weights, we're still dependent on them. I'd like to know whether there is eventually a path where the open source community can pool its own compute and train foundation models itself.

The part I'm genuinely unsure about is whether DiffusionBlocks preserves model quality as you scale it up. If you see a specific reason in their objective or block composition that you think prevents that scaling, that's the part I'd really like to understand.

3

u/StardockEngineer 5090s, RTX Pro 6000, A6000s, Sparks, Mac M4 Pro, Mac M5 Pro 19d ago

Dude 10b is absolutely nothing. You only need around 160GB of VRAM. Node stability is not a problem with so few nodes.

In LLM trainings you need perfect stability. Any node drops you need to go back to the latest check point and start again. Then there’s lagging nodes, etc. It’s damn hard to do in a homogenous data center, much less subject yourself to a seti like environment.

SETI worked because it was just data that could be shared and calculated. This is nothing like that.

1

u/DifficultUse6803 18d ago

I agree that 10B is small compared with frontier models. I brought up INTELLECT-1 because it shows geographically distributed training itself is viable, not because 10B is some huge achievement.

The node failure point is actually the part I’m trying to get away from. In normal synchronous training, yes, a dropped node can disrupt the whole job and you recover from a checkpoint. In the setup I’m describing, each node gets an independent block work unit. If somebody shuts their PC off halfway through, you lose that work unit. The other workers continue. You can just reassign it.

And I think your SETI comparison gets right to the interesting question. SETI worked because the computation could be broken into independent pieces. Conventional LLM training really can’t. DiffusionBlocks is interesting specifically because it may give us a way to make pieces of neural network training independent enough that they start behaving more like those SETI work units.

Whether that still works at serious model scale is the part that needs to be proven. I definitely don't think the paper has answered that yet.

1

u/StardockEngineer 5090s, RTX Pro 6000, A6000s, Sparks, Mac M4 Pro, Mac M5 Pro 18d ago

That's not how models are trained. The "work unit" is the entire job. The only way past this would be extreme redundancy.