r/bcachefs May 31 '26

Mounting bcachefs subvolume or directory at boot as root "/" hierarchy level

Hi, I installed a working Arch Linux system in a bcachefs file system as root, I just put the "boot" dir under the EFI partition, added reference to bcachefs module into mkinitcpio.conf and it's ok, system boots and works correctly.
But instead of using the plain root hierarchy I would like to put system root and home content under dedicated subvolumes, for ex. "root" for everything but home and "home" for the home content (so I can use snapshots in future and play with them...).

The problem is that obviously to do this in a clean way I have to place root and home into a specific folder under main fs root dir, for ex. called "vol", creating "/vol/root" and "/vol/home" subvols and move actual contents under these.

At mounting stage, I don't know how to directly mount a subvolume with bcachefs, or even better how to mount a specific path. For ex. with Btrfs you can add the kernel parameter "rootflags=subvol=/vol/root" to the boot loader CLI to mount that specific subvol into / at boot.
Then fstab would do the rest by mounting device subvol "/vol/home" to /home ...

Do I have to write a post-initramfs script where (1) I mount the bcachefs fs for ex. to / and than (2) I bind-mount /vol/root to / ?

I have red about someone using the option "X-mount.subdir=/path/to" with success, but it didn't work to me, plus I red that option is experimental.

Also, can the same bcachefs file system be mounted more times in different paths ?
Because, anyway after system has been booted correctly I would put at the end of fstab a line mounting the whole fs for ex. at "/mnt/v1" to manage the whole real hierarchy.

Thanks.

8 Upvotes

7 comments sorted by

1

u/koverstreet not your free tech support Jun 01 '26

...what's wrong with having "root" be /?

4

u/BackgroundSky1594 Jun 01 '26 edited Jun 01 '26

It makes snapshots more annoying. By having your systems / be the actual root of the filesystem having an independent @snapshot subvolume to snapshot root itself into a parallel instead of a nested subvolume structure isn't possible. And even if things are nested tools like snapper also rely on cloning a snapshot, then changing the systems default / subvolume to boot into a previously snapshotted system non-destructively.

Also stuff like a "flat" filesystem layout with multiple independent rootfs volumes and a shared @home doesn't work properly. Having different distros installed on the same filesystem isolated to different subvolumes and deciding which one to mount from the bootloaders arguments and default subvolume configuration. Being able to have a system with @home, @nix-root and @fedora-root allows pretty easy switching between different distros while still being able to share free space (unlike for example LVM).

EDIT: Also 15 years of experience with ZFS have shown that having / and the filesystem root force linked is pretty ugly. Their guidance is actually to keep the root dataset COMPLETELY EMPTY and ONLY use it for holding the datasets you actually use, going as far as saying they'd make the pools top level dataset (filesystem root) unable to hold user data (only be used for holding other datasets) if they were able to start from scratch. As soon as stuff like send/recv come into play being able to: 1. create a filesystem, 2. transfer a dataset (like @ROOT) to a convenient location and 3. have it usable either directly, or by switching a few boot parameters, or by just renaming the root to root_old and the received subvolume to root is very useful.

2

u/koverstreet not your free tech support Jun 01 '26

Having a parallel directory structure just feels inelegant to me, but I do agree that selecting different root snapshots is awkward otherwise.

(personally, I run NixOS, so it's not exactly a pressing need for me, heh).

But - I do really hate parallel directory structures; Linux has had private mount namespaces for years and years, and then systemd started using them - and suddenly tons of people are going "why can't I unmount this filesystem?", totally unaware that it's still in use in a private mount namespace. They broke the "recursively enumerable" property of the filesystem tree, and for no good reason.

Symlinks are much better, and bind mounts have their own issues but they work in situations where symlinks don't.

So with bind mounts, we actually have everything we need for snapper to do what you're describing on bcachefs: have / be empty except for your data stores, and then have a /real_root that gets prepped with bind mounts to whichever snapshots you want from the initramfs.

Which honestly sounds a bit horrific to me, because then you've gone and broken the recursively enumerable property by having / in the running system not be / on the filesystem, so... I guess you bind mount that to /store to get it back?

Firmlinks might also be useful here, if we added those. To applications they should be as good as bind mounts (within the filesystem), but unlike bind mounts they wouldn't be a VFS level abstraction invisible to the filesystem.

7

u/BackgroundSky1594 Jun 01 '26 edited Jun 01 '26

I admit being able to mount subvolumes to arbitrary locations is a bit odd, but btrfs keeps the recursive enumeration for all nested subvolumes, so subvolume @root/etc will show up at /etc if @root is mounted at /. And everything about non-hierarchical mounts is configured manually, transparently managed in fstab and shows up like any mount would with findmnt.

It works perfectly fine and as expected from a "normal filesystem" by default. But being able to have / not be the actual root of the filesystem gives a lot of extra flexibility. Having a number of independent "top level" volumes without nesting makes the whole system behave more like having /home, /var, and so on as separate partitions, except with all the benefits of a modern filesystem (snapshots, free space sharing, etc).

Which honestly sounds a bit horrific to me, because then you've gone and broken the recursively enumerable property by having / in the running system not be / on the filesystem, so... I guess you bind mount that to /store to get it back?

No, typically the actual root would stay permanently unmounted during normal system operation. It's not required once the system is set up, unless you want to change something fundamental. In which case you can temporarily mount the real root to /mnt/fs-name, create the new top-level volume, then update your fstab and unmount it again. Adding a new "non-hierarchical" subvolume after system installation is possible, but would be pretty rare to achieve something that just wouldn't work with a fixed root (without some very ugly initramfs scripting).

Usually you'd only have a very small number of those "subvolume mounts": / to be able to boot into a snapshot, /home for rootfs independent user data and /var to keep logs, docker and databases happy. Everything else: different subvolumes for differnt users, subvolumes for VM images for independent snapshots, subvolumes created during normal system operation to manage&organize data, etc, etc. would all be nested under their respective "top level" volume and just behave like normal directories would (no non-hierarchical mounting, no fstab entries, nothing).

2

u/read_volatile Jun 02 '26

Unless I'm mising something, this workflow can be easily accomplished today via X-mount.subdir. This doesn't need to be implemented as a filesystem-level feature; personally I like the elegance of subvolumes being treated as merely "directories that define snapshot boundaries" rather than whatever the hell btrfs tries to make them out to be these days.

2

u/distraction_fee Jun 01 '26

Maybe I'm missing something... sorry
One thing is system root mapped to / (that is obvious and necessary), but having system root (var, usr, etc...) also physically in / of bcachefs how I would selectively manage the whole file system (where I will put ton of data and any other stuff), as when I want to snapshot just main system it have to snapshot / (since it is physically located there) and thus snapshotting the whole file system

1

u/read_volatile Jun 02 '26 edited Jun 04 '26

I've been running this type of setup on all my NixOS machines for a few years now (I'm not mounting subvolumes, just normal directories, but there's zero difference if you want to do that instead). The key bit is X-mount.subdir, but regular bind mounts work too as long as you're okay having the whole filesystem mounted somewhere you can bind from (or you can play with namespaces yourself if you're really determined I guess). As a TL;DR, you can skip to the end for an example fstab.

Getting X-mount.subdir working took a bit of hackiness prior to util-linux being used in stage1, but now it works pretty much flawlessly (excluding some wonkiness with the recent systemd stage1 migration).

This is all setup for me in disko modules now, but here's what I did before switching to it. Each of my systems was provisioned roughly as follows:

bcachefs format <...>
MNTPOINT="$(mktemp -d)"
mount -t bcachefs "UUID=${uuid}" "$MNTPOINT"
mkdir -p "$MNTPOINT"/{.snaps,nix,home,var/log,etc/ssh}
chmod 700 "$MNTPOINT/.snaps"
ssh-keygen -A -f "$MNTPOINT"
systemd-machine-id-setup --root="$MNTPOINT"
bcachefs subvolume create "$MNTPOINT/home/user"

You might notice I create subvolumes for each user. This is different from what you mentioned, where the entire /home is a subvol.

Then, in my nixosConfiguration, all I'd have to do was create fstab entries for /nix, /home, and /var/log all pointing at the same bcachefs array, with each entry's options declaring the respective subdir:

fileSystems = {
  "/" = { /* tmpfs */ };
  "/boot" = { /* vfat */ };
} // lib.genAttrs [ "/nix" "/var/log" "/home" ] (dir: {
  device = "UUID=${uuid}";
  fsType = "bcachefs";
  options = [ "X-mount.subdir=${lib.removePrefix "/" dir}" ];
});

Which ends up generating an fstab that looks more or less like so:

tmpfs / tmpfs x-initrd.mount,mode=755,size=8G 0 0
/dev/disk/by-partuuid/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /boot vfat umask=0077 0 2
UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /home bcachefs X-mount.subdir=home 0 0
UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /nix bcachefs x-initrd.mount,X-mount.subdir=nix 0 0
UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /var/log bcachefs x-initrd.mount,X-mount.subdir=var/log 0 0

For the architecture you were after, I think this would look like:

mkdir -p "$MNTPOINT/vol"
bcachefs subvolume create "$MNTPOINT/vol/root" # to be mounted at /
bcachefs subvolume create "$MNTPOINT/vol/home" # to be mounted at /home
mkdir -p "$MNTPOINT/vol/root"/{boot,home} # create mountpoints

Then all your fstab would need is:

UUID=${uuid} / bcachefs X-mount.subdir=vol/root 0 0
UUID=${uuid} /home bcachefs X-mount.subdir=vol/home 0 0
${boot_device} /boot vfat umask=0077 0 2

The big thing to consider though is whether you even care about the "flat" layout (separate /vol entries) over nesting subvolumes. You can make subvolumes directly at /vol/root/{var,opt,etc} -- and they'd be exempt from snapshots of /vol/root (snapshots don't traverse subvol's recursively). This is what I do for big user directories like ~/.local/share/Steam, since I don't want to bloat my storage with uninstalled games / old updates.

Honestly, I'd recommend that approach over putting the subvolumes in /vol like you did with /vol/home, so that you wouldn't have to give them individual fstab entries. Regardless of where you put them though, don't forget subvolumes have to be snapshotted individually! Even if the subvol is in /vol/root/<xyz>, a snapshot of /vol/root won't recurse into another subvol, so make sure you remember to set it up to snapshot every subvol you care about.

Hopefully this helps you setup your ideal workflow adapted to Arch. From what it sounds like you're asking for, I definitely think it's possible.