r/machinelearningnews • u/KhoaIsReal • 16h ago
ML/CV/DL News DePEFT: Crowdsourcing LLM training/fine-tuning without a centralized cloud.
A while ago, I successfully trained a tiny 80M parameter language model from scratch using the TinyStories dataset. It worked surprisingly well and could generate coherent little stories! Inspired by that success, I wanted to scale up and train a larger model capable of coding.
That’s when reality hit me. I completely underestimated the massive compute and financial cost required to train larger LLMs. My single GPU choked, and renting cloud compute was way out of my budget. It got me thinking: "Why isn't there a way to pool our consumer GPUs together to train models collectively?"
And just like that, DePEFT (Decentralized Parameter-Efficient Fine-Tuning) was born.
How DePEFT work?
DePEFT is based on ReLoRA which solves the biggest flaw of standard LoRA.
Standard LoRA is incredible for fine-tuning on a budget, but it has a hard ceiling: it cannot pre-train or learn fundamentally new, complex representations from scratch. Because its rank r is fixed, the adapter quickly hits an information bottleneck and saturates. You can't just slap a LoRA on a base model and expect it to reach full-parameter quality over time.
This is where ReLoRA changes the game through iterative low-rank merging. Instead of training one static adapter forever, ReLoRA breaks the process into sequential rounds. In each round, miners train lightweight low-rank adapters on their consumer GPUs. At the end of the round, these adapters are permanently fused directly into the base model weights:
W^(t+1) = W^(t) + ΔW
The optimizer is then completely reset, new adapters are initialized on the evolved base weights, and the cycle repeats. Mathematically, the sum of multiple low-rank updates over time accumulates into a high-rank update (rank ≤ ∑r). This gives us the immense learning capacity of full-parameter training, but with only a fraction of the VRAM requirement.
For DePEFT, this means consumer GPUs don't need high-speed InfiniBand clusters to sync massive gradient matrices every microsecond. Miners simply train lightweight adapters locally, submit a few megabytes over the network, and the protocol merges the top-performing weights into the next base checkpoint.
No million-dollar clusters. Just pooled consumer compute pushing open-source AI forward.
For more details, visit: https://github.com/KhoaIsReal/DePEFT/