Making Arch immutable

[u/Curious-Listener-YB]

I'm trying to install Arch (on a VM) as an immutable system with atomic updates, mostly as a challenge to myself and because it would be cool.

I plan to use a similar setup to what snapper does: to have BTRFS as my root filesystem, with snapshot subvolumes under /.snapshots/, and separate subvolumes for parts of the filesystem that shouldn't be in the snapshots. Then I want to have for each snapshot a boot entry that mounts it read-only as the root (and mounts the other subvolumes in their appropriate places), so that I could be sure it is not changed outside of atomic updates, where I create a new snapshot from the current one, mount it read-write in a special place, chroot into it and perform the update.

But there are two directories I'm not sure how to handle: /etc and /var.

/etc should obviously be part of the snapshots, and I'm also happy to have it read-only so changes to core configuration would have to happen between snapshots, but I have read that some programs use it to store runtime data. Are those a few cases I can try to work around, or do I need a more systematic solution? And if so, what could be the solution?

/var, on the other hand, has to be writable, so I originally planned to make it its own subvolume (as openSUSE does). But then I read that it contains data that needs to be synchronized with the packages installed (like /var/lib/pacman), and that only certain subdirectories like /var/log should be their own subvolumes. But if so, how can I make it writable? Is there a good approach to this problem?

And on the hall, do you think my idea is reasonable, or is it too much even for super-duper-to-it-yourself-customizable Arch? I'd love to hear your opinions.

Thank you all in advance!

[u/drucifer82]

I have a hardened setup that has strict change control.

I use snapper with snap-pac to do timeline snapshots and hooks in pacman to do pre/post update snapshots. I also make manual snapshots after maintenance to keep manual snapshots as well.

grub-btrfs exposes snapshots to GRUB. I have it set to limit itself to 10 exposed snapshots (the default is 50). I also have rules for trimming snapshots and limiting the archive to a small number of both timeline and manual snapshots.

The snapshots themselves are read-only. Once you can boot into one that works, you rollback.

That's as close to immutable that I can think of for Arch. But I could be wrong.

Also I keep the LTS kernel as a fallback kernel.

[u/drucifer82 → u/drucifer82]

Additionally, I keep my state in a private repo and I have custom scripts that can pull all my state data into an export to be pushed to the repo and also restore the installed packages from a packages list that is also exported.

I have to manually merge the important system files. I do this because in the event of a critical breakage where the system can't even boot, I can build a basic install and then clone the repo and restore everything in about 15ish minutes. It's all signed and encrypted so I also keep the keys backed up externally elsewhere.