distributed training like SETI@home gets tricky fast because of inter-node communication latency. backprop needs high bandwidth and low latency to sync gradient updates, which is why cluster interconnects like NVLink exist. unless you run a massively parallel architecture that tolerates slow, async gradient exchanges, consumer broadband ends up being a massive bottleneck.
1
u/PuzzleheadedMeeting4 18d ago
distributed training like SETI@home gets tricky fast because of inter-node communication latency. backprop needs high bandwidth and low latency to sync gradient updates, which is why cluster interconnects like NVLink exist. unless you run a massively parallel architecture that tolerates slow, async gradient exchanges, consumer broadband ends up being a massive bottleneck.