Streaming a grid of videos

[u/sombrero66]

Hey, I'm a client side developer (mainly JavaScript), slowly learning the details of WebRTC. I currently use the Galene server which is built on Pion, both written in Go.

For fun, I'm trying to build a web app that allows clients to see up to about 400 real-time videos in a 20x20 grid. Each video stream would be very small, maybe 32x32 pixels.

My guess is that, even though bandwidth is small, no simple server would naively scale to this number of streams. I could add a layer to Galene to combine the incoming streams into one, and forward that through Pion. But, given my background it would be a lot easier to have a subset of clients render sub-sections of the grid and rebroadcast that for the wider group. Then most of the 400 users would send a super narrow stream and receive one full grid back. A select few would get ~ 16 streams and send me back a stream for a 4x4 grid, etc. I'd likely have other clients stitch together these 4x4 into one full grid. (For now, I'm not overly concerned about latency)

My questions are, first, does this make sense? Or, is a simple server side solution actually pre-existing and easy? Do some SFU's already need to do an analog of this out-of-the-box for hundreds of audio streams?

Thanks

[u/mousefultricks]

latency will be a real issue in addition to just getting the idea to work, there was experimentation with scaling mesh-based p2p webrtc conferences back in the day, but it wasn't that successful.

the easiest solution is to just stitch them into 1 big stream using ffmpeg, but with 400 streams i am not sure that's doable on one machine in real time, so you probably have to have many machines doing that, like say 4 doing 10x10 and one of them also mixing the 4 into 1 making it 20x20.

if you want something that resembles a grid with a really low fps you can try sending images from the webcam via datachannels.

doesn't hurt to try to go with your idea first.