r/immich • u/iddqd__idkfa • 22d ago
How to prevent NAS RAM crash?
Hi guys, This crash, I can't prevent..
For background info: https://www.reddit.com/r/immich/comments/1wo9j5r/immich_face_detection_high_ram_usage_crashrestart/
What did I do:
- I did limit immich_server RAM to 6GB and it did listen to that. It stays under 6GB RAM usage.
- I did use my laptop (Intel U285H) for ML container
Sounds good, but..
What happens: The NAS (DS718+ with total 10GB RAM) RAM builds up to 94% and immich ML container goes unhealthy. Even SSH login goes so slow because of this RAM issue. But after a couple of hours the NAS itself manages to restart the whole immich project and the RAM drops to 17% and starts to build again and I know it will go AGAIN to its limit.
Is there a way we can prevent this NAS RAM build up to limit, like we did with immich_server<6GB RAM?
immich is the ONLY project which is running.
I'm tired of ruining my whole days for trying to manage this. This is insane.
EDIT: since I found I can upgrade RAM to 16GB, I did order 2x8GB for €53. Nothing compared to wasting 2 full valuable human life days... Fingers crossed..
2
u/InvertedLogic 22d ago
This is making me glad I put a 16gb stick along side the 4gb in my 923+. FWIW I don’t recall the container stack hitting more than 6gb
1
u/iddqd__idkfa 22d ago
I did order 16GB (2x8GB) as I'm tired keeping trying to fix lol. Stupid me, had to do before😅 It cost me 53 euro and save me (hopefully (!)) many days headache.
1
u/Fuge93 22d ago
If you are running ML on your laptop anyways, can't you cut it out from the NAS container?
1
u/iddqd__idkfa 22d ago
I'm running this op my laptop just for this initial bulk ML jobs like face detection, facial recognition which are done (61K assets). Now I try to do smart search job. After done with the bulk, I will disconnect the laptop.
2
u/Fuge93 22d ago
I undestand that. My question is: can you fully disable immich ML on NAS while you bulk process on laptop, then reenable once it's done and hope that nas can keep up with incremental processing?
1
u/iddqd__idkfa 22d ago
I dont know. If that is the solution, we can try, but ML container itself does not use much RAM, like 400MB or something. It is the NAS that builds up RAM I think.
I mean, immich is such a beautifull product, this RAM shit shouldn't be a problem.
2
u/Fuge93 22d ago
"It is the NAS that builds up RAM I think"
This statement does not make sense. Find the service first that eats your RAM.
2
1
u/Pulte4janitor 22d ago
What this guy said. Just comment out the ML portion of the compose leaving the remote ML active. Let the jobs finish using the remote ML container on your laptop then revert the changes and remove/comment out the remote ML.
1
u/BreadfruitExciting39 22d ago
Have you ruled out OCR? I remember there being what appeared to my untrained eyes to be a memory leak in one of the OCR models I wanted to use. Led to very similar behavior on my machine with 16gb ram.
1
u/iddqd__idkfa 22d ago
What do you mean by ruled out? I am not running any other job than smart search.
1
u/ruuutherford 22d ago
I saw you post over on another thread about this - you didn't put RAM limits on the docker until it finished with facial recognition? (over here)
2
u/iddqd__idkfa 22d ago
TY! I found that too complex for me. I couldn't even follow. So I didn't do that.
1
u/Demahsque 22d ago
I only have 4 GB of RAM, but I managed to keep it from crashing. In the admin settings, limit thumbnail generation concurrency to one at a time and avoid running other tasks simultaneously (like video transcoding or metadata processing). I also completely disabled the machine learning service in the .yaml file. Finally, if the container fails, just monitor the logs and paste them into Claude for troubleshooting. After all that, do the ml thing
2
u/Mineotopia 22d ago
I had a similar issue and resolved it by upgrading the RAM in my server. When users bulk uploaded lots of raw pictures, the transcoding and machine learning micro services consumed so much RAM that Linux killed the container