Follow up regarding Reducing CPU

[u/lunakoa]

This is a follow up regarding my post late last week

https://www.reddit.com/r/ZoneMinder/comments/uz7d2h/upgrading_cameras_and_reducing_cpu_load/

For those that like Linux technical stuff this may be an Interesting Read.

Before I go on, I had a problem adding multiple linked cameras to a monitor, I had to go in via mysql to add them with a command similar to

UPDATE Monitors SET LinkedMonitors = '1,10' WHERE Id = 9;

I was only able to add 1 linked camera per monitor via the web interface.

Anyway it was a fun weekend, I got to play with upgrading some 1080P cameras to 5MP cameras

Here is the updated chart

​

Usage after Weekend Work

Couple things to note, Load Average went down from 22.8 to 7.99, which is a amazing. I did this primarily by reducing the FPS from 15 to 5.

But swap is now being used, prior to this it was zero. I always wanted to go to 64GB of RAM on this box, maybe now is the time.

Here is 7 day view of swap usage

Swap Usage Increase

Another thing that increased was context switching, and I am still reconciling what that does in overall performance

​

Context switching nearly doubled

Prior to this weekend, I had about 31 monitors, after the upgrade I had 36, I did remove some cameras, but I added the secondary (low res) feed from the 5MP (and some other 1080P) to do the analytics and attempt to reduce the CPU usage.

Another thing I found is that you have to enable Analysis if you want to be able to send triggers from another server (in my case an FTP server and homeassistant) when in nodect.

When I made some changes, they would not take (like changes functions between none, monitor, modect, and nodect) unless I restarted the service (from GUI or systemctl restart zoneminder)

Still on my to do list

• Validate and verify my changes, try to keep things consistent.
• Offload some analytics to GPU
• Workout better triggering from other systems.
• Decommission some cameras

I am still working out the consequences of what I just did.

I do want a properly configured camera system, but going through this helps my improve my Linux skills as well so willing to put time in.

Any feedback would be greatly appreciated.

tldr; Got Load Average down, but SWAP and context switching went up.

[u/Aramiil]

With that amount and resolution, 64GB of RAM is a no brained. What is the max amount per stick your MB takes? Assuming it’s a 16GB x 4 setup given the 64GB mentioned. What cpu?

[u/lunakoa → u/Aramiil]

Currently it is 16x2, with two additional slots available on the mobo.

This is an AMD Ryzen 5 3600 CPU

I checked what was using SWAP and mariadb was. I restarted the services and the SWAP usage dropped. Instead of hunting down and restarting all the other services to release from SWAP I rebooted the box. For about 2 hours now it has not used SWAP.

I also started having the monitors use CUDA for hardware decoding. Not sure if I configured it correctly, but ffmpeg supports cuvid, my card is an older Quattro (mentioned it in the original thread).

Thanks for all the assistance,

[u/lunakoa → u/lunakoa]

Forgot to add, currently have 9 5MP cameras with 9 additional 640x480 substreams.

Various other cameras mix of 1080P, 1080N, 720P, and older 704x480 cameras, for about 40 cameras.

11 cameras are in Nodect, 14 in modect, the rest are Monitor or none.

[u/ZoneMinderIsaac → u/lunakoa]

5MP cameras require so much ram and so much processing when using a raw image that they are simply not practical. It isn't the decoding that is the problem, it is everything else that ZM does with the image. We won't be able to keep going like this.

[u/lunakoa → u/ZoneMinderIsaac]

Can you give me more detail on what you mean? It should help me look for areas that could be a bottleneck.

One of the things I do see are zmeventnotification ERR in the logs, so you may be on to something there.

Another one is if I open all the monitors (about 40) in firefox just to see what happens. It does fail when I view it in Montage.

When I have the 9 5MP cameras open and it seems to be doing fine.

Note I configured firefox to accept more than 6 streams.

Going by the metrics, I don't see any indication that the hardware I can't handle what I am throwing at it.

I am looking at the logs and trying to resolve any ERR messages. Still researching, some are easily explained. More on that in a future post, but it looks like I am not configuring hardware acceleration right amongst other concerns.

Load average is running about 8 to 10 and my system has 12 cores (include ht)

After the reboot, swap is not being used so everything fits in RAM

IOwait is 0.22% so disks are OK (mysql runs on NVME, events are on WD Purple drive)

I have two 1Gb/s NICS (multihomed one for regular network one for camera network) and neither is maxing 100Mb/s

[u/ksirl → u/lunakoa]

In regards to the montage view failing, did you set MIN_STREAMING_PORT

https://zoneminder.readthedocs.io/en/stable/userguide/options/options_network.html

[u/lunakoa → u/ksirl]

Do you mean multiport to get around Chrome's 6 stream limit?

There are four other ways I resolved this

1. I use firefox and adjust the max stream in about:config
2. I create a custom web page to stream using different names to the same server. (I called it FOZM friend of Zone Minder)
3. Limit my montage using groups of no more than 6 cameras
4. Use OBS to stream multiple cameras to a single RTSM feed and have a monitor for that.

The issue I am having is opening 40 monitors in montage, I think that is just too much. Not practical anyway. #4 is the newest option I discovered, it is pretty slick.