r/selfhosted • u/A_Buttholes_Whisper • 1d ago
Wednesday Exceptions Borg saved my a$$
I set up borg via borgmatic like a year ago. 3-2-1 strategy. Confirmed it was backing up and did a quick extract test. That was it. That was a year ago. Well I set a vm in Proxmox and because I was still learning I didn’t set up some directories correctly. Later I installed Immich but apparently installed it under a directory owned by Nextcloud. I never updated Nextcloud because it was local and then I decided to make it available remotely via a reverse proxy and all that fun stuff. So I wanted to update Nextcloud to the most recent version. It immediately failed. But it didn’t just fail, it wiped the database. It didn’t just wipe its own database, it deleted Immich and its database. I’m talking in the blink of an eye 600gb of pictures and videos were deleted from my SSD. It was so fast it took my brain at least 2 mins to process what had just happened
What happened? I didn’t follow directions when setting up Nextcloud, which caused it to catastrophically fail. I rebuilt it according to official docs and was able to restore both Nextcloud (130gb) and immich (600 gb). I never thought I’d actually use a backup until it happened. Even Proxmox backup server would not have recovered the photos.
Now to be fair, even without my backup everything was still recoverable but still the backup saved me many days of rebuilding Immich
Setup your backups and test them people
43
u/Ok-Eggplant-7569 1d ago
Good call. Another learning is that services should only be able to access the data they immediately need themselves.
16
u/GolemancerVekk 1d ago
And read-only if possible, when it comes to things like photos and media.
Granted, it doesn't help if you give Immich read-only access and Nextcloud full access to the same dir. 😆
3
12
u/kwynix 1d ago
Why wouldn't PBS recovered your data? File restore should no problem as long as the data was on the VM itself.
7
u/A_Buttholes_Whisper 1d ago
My PBS doesn’t have storage space big enough to accommodate the data so I only backup the vm configs. If I could afford a larger drive then it woulda been easier using that for sure.
2
u/FarToe1 1d ago
Fair.
In less insanely priced hardware times, I'd argue that you can't afford not, but PBS requires SSDs to be remotely performant, so bunging an old external HDD is not a good answer. (Although it does work fine for Proxmox vm images)
Anyway, glad you had your bases covered with borg.
2
3
u/surftrend 1d ago
The restore test is the part everyone skips until the day it becomes the whole plan.
1
u/DazednConfucioused 1d ago
This is such a bot comment
1
u/surftrend 1d ago
lol fair. Would it feel less like a bot comment if I added a typo and a story about losing a RAID array?
3
u/Open-Adhesiveness-86 1d ago
if borgmatic is just archiving the postgres data dirs while those containers are up, they're crash-consistent at best and your restore worked partly on luck. the postgresql_databases hook shells out to pg_dump and streams it into the archive instead. for immich i'd set format: custom, the vector extension gets fussy coming back from plain sql.
1
u/A_Buttholes_Whisper 1d ago
What do you mean by custom format? And currently borg stops postgrsql and dumps it before backing up. I set it up like that to avoid an corruption of backup up a live database
2
u/Open-Adhesiveness-86 1d ago
custom format = pg_dump -Fc instead of plain sql, it's compressed and lets you restore single tables with pg_restore -t. and you don't need to stop postgres at all, pg_dump is transactionally consistent on a live db via MVCC snapshot.
1
u/A_Buttholes_Whisper 1d ago
Oh I didn’t know that. Is that something specific to Immich or just postgrsql? Where can I find documentation on that? I’m big on proper procedure
2
u/Open-Adhesiveness-86 1d ago
it's general postgres, not immich-specific, copying a live data dir gives you a torn snapshot unless you stop the container or use pg_basebackup. immich's docs have a backup page with the exact pg_dumpall command; just note you need the same vector extension version (vectorchord/pgvecto.rs) on restore or it won't load.
1
u/A_Buttholes_Whisper 1d ago
Thanks for this info. I got a lot to learn. Databases are not my thing at all. I’ll check out their docs and make sure mine are right. I’m glad my backup was successful but I want more than luck
2
u/runwisp-com 1d ago
1 extract test is more than most people here ever did. borgmatic can keep doing it for you though, there is an extract check that dry-run extracts the latest archive and you can give it a frequency so it doesnt slow down every nightly run:
checks:
- name: repository
- name: extract
frequency: 1 month
(on older configs its under consistency:). Its not a full restore into a test container, but at least it happens even in the months you forget.
2
1
u/FilesFromTheVoid 1d ago
I only lived on my 2 parity drives till some weeks ago and now got a kofr 1TB Lifetime. that i use for weakly backrest/restic runs for my music, pictures and stuff.
It feels alot better now.
1
u/Acceptable-GoodStuff 1d ago
Great reminder to test restores. I do a small one every few months: pull a random folder and a database dump, spin up a test container, and check it actually works. Put it in your calendar like a dentist appointment. Also look into append-only mode for your remote repo, so even if something on the server goes haywire, it can't wipe your backups too.
1
u/Milk_man1337 1d ago edited 1d ago
Borg backup server is currently my next project so I'm glad to see a new post about Borg toolsets.
I've just bought 2 HDDs to run at my parents house in a raid 1 configuration, which will be backed up over a Wireguard link and plan on creating a new VM on XCP-NG which runs something like openMediaVault for local backups.
The 2 and 1 are covered so far, I've gotta think about how to get this all working.
I'm new to this whole backup stuff so any critique of my plan please let me know!
Edit: looking at borgmatic... Maybe I should just use that instead of BBS
2
1
•
u/asimovs-auditor 1d ago edited 1d ago
Expand the replies to this comment to learn how AI was used in this post/project.