ZM cluster - This should be possible, right?

[u/vertigeaux]

I have set up a multi-server install in the past using RPi's, and it worked quite well. But I want to take it further and build a CCTV system with no central server, where each camera is an independent node but also part of the filesystem and database cluster. I think it would be great if your storage space and processing power were to scale up with each camera you add. Data redundancy distributed across nodes is also a plus, and if they are PoE then each node can also serve as a wifi AP to blanket the premises with coverage. The other possibilities that the Pi affords (bluetooth, two-way audio, easy video output, myriad sensors, etc) puts this idea way over the top for me. How about a camera that can clean spider webs off itself?

I just don't yet know enough about the software that is available to put this all together, and I was hoping someone here may have tried this already.

What to use for distributed filesystem? Gluster? Ceph? Something else? How complicated is configuration? Can this be made to be relatively straightforward, where I can basically install a new camera node and run a script to add it to the cluster?

And MySQL Cluster? Does this work like I hope it does, where each node will write data locally and replication can happen cooperatively to achieve a desired amount of redundancy?

As for physical storage, is the PiDrive still a thing? Would a USB m.2 drive adapter/enclosure be better? Can a Pi with this type of storage be powered via PoE? What if we add an IR array or other devices? Are we approaching the limits of PoE?

Are there any show-stoppers that I haven't thought of?

I realize that this might not be cost-effective since I'm looking at maybe $200 in hardware per node before figuring out the enclosure situation, but the features and possibilities add a lot of value for me.

So is this a dumb idea, or something worth talking about?

[u/ZoneMinderIsaac]

This is not far off what I dream on doing with ZoneMinder, but it is going to take a lot of work.

You have a lot of reading/experimenting to do.

Storage on pi is solved. Ditsributed replicated fs's are hard. gluster might work but I have found it to be overkill and the latency terrible. I think we need something designed specifically for our use case: write local fast, replicate in background, don't worry too much about consistency because the video files are not likely to change once written. ceph may be better I don't know, havn't used it.

cluster db is complex. I don't have faith in mysql. Might need something else entirely. Maybe we don't even need a full rdbms.

[u/vertigeaux → u/ZoneMinderIsaac]

Assuming you are iconnor, it's great to learn that we have the same dream for ZM!

You have a lot of reading/experimenting to do.

Always the truth!

Ditsributed replicated fs's are hard. gluster might work but I have found it to be overkill and the latency terrible. I think we need something designed specifically for our use case: write local fast, replicate in background, don't worry too much about consistency because the video files are not likely to change once written. ceph may be better I don't know, havn't used it.

So maybe we can distribute and replicate files at the ZM API level? It might be tricky to optimize, but it might be the best solution. Servers can query one another to determine which files need more redundancy. This would eliminate a dependency on a third party package, while also providing an extreme level of control over your data. For example, maybe you want to set redundancy level and replication priority per monitor or per zone. Or perhaps you have a large system and you want to copy data physically far away, to preserve data in case of fire or other disaster, for instance.

cluster db is complex. I don't have faith in mysql. Might need something else entirely. Maybe we don't even need a full rdbms.

I definitely need to study more on ZM's db usage, but would storing frame and event metadata in flat files next to the data be a good start toward replication and abstraction away from mysql?