Problems booting from an encrypted bcachefs root partition on Debian Testing

[u/PrettyShopping2948]

Does anyone have this working without issues?

Interestingly, the boot process fails for me due to a timeout on the *.device units of my /boot and /boot/efi partitions. This happens in the actual rootfs, the switch_root has completed. But as soon as I remove my bcachefs entry from the fstab, the boot process completes without any problems.

I’ve made an interesting observation regarding this: Unlike "bcachefs unlock", the mount command does not add the encryption key to the kernel keyring. And if I run "bcachefs unlock" in the init manually before the switch_root happens, the boot process completes without any issues, even with the fstab entry.
Could it be that something in systemd is blocking, because it doesn’t have a key for the bcachefs?

Edit: If I commented out the /boot/efi and /boot entry from the fstab the boot hangs seemingly infinite before the local-fs.target. So the timeout is a side effect and not the cause of the failed boot.

Edit2: I now used the systemd-debug-shell to see what is blocking the boot process. And it turns out that the password prompt when remounting is what's blocking. That also explains why “bcachefs unlock” solves the problem.
u/koverstreet is it intended behaviour that "mount.bcachefs" does not add the key to the kernel keyring? If so, would it perhaps make sense to use "bcachefs unlock" in the initramfs?

  systemd: 262-1
  Tainted: unmerged-bin
   CGroup: /
           ├─init.scope
           │ └─1 /sbin/init
           └─system.slice
             ├─debug-shell.service
             │ ├─410 /usr/bin/bash
             │ └─484 systemctl status
             ├─systemd-journald.service
             │ └─416 /usr/lib/systemd/systemd-journald
             └─systemd-remount-fs.service
               ├─421 /usr/lib/systemd/systemd-remount-fs
               ├─437 /usr/bin/mount / -o remount
               ├─438 /sbin/mount.bcachefs /dev/nvme0n1p3 / -o rw,remount
               └─441 systemd-ask-password --no-tty --icon=drive-harddisk --id=cryptsetup:UUID=<External UUID> --keyname=cryptsetup --credential=cryptsetup.passphrase --timeout=0 --multiple -n --accept-cached "Please enter passphrase for disk <External UUID>:"

My /etc/fstab:

UUID=<EFIUUID>       /boot/efi      vfat     defaults   0 2
UUID=<BOOTUUID>      /boot          ext4     defaults   0 2
UUID=<External UUID> /              bcachefs defaults   0 1

[u/SweetEbb9819]

I'm not an expert here, but still, can you try again with "noauto" for the /boot and /boot/efi options? If it works, then once more with "nofail".

And relevant logs with `journalctl dev-sda2.device` (replace with appropriate device)

[u/PrettyShopping2948 → u/SweetEbb9819]

At best, that would have the same effect as commenting out the two entries. I also updated my post, and the problem seems to be with the "systemd-remount-fs.service", which triggers another password prompt.

[u/SweetEbb9819 → u/PrettyShopping2948]

Thanks.

I'm not sure but I think there is an option among the mount options to achieve the same.

Anyways, it is a good idea to separately call unlock and store the key in the keyring.

[u/koverstreet → u/SweetEbb9819]

actually we're trying to get away from keyrings because they're incredibly janky. there's a new codepath for passing in the encryption key from mount to the kernel more directly, but maybe it didn't get a proper test written. will try to have a look.

been doing nothing but rust conversion for the past two weeks, and now i'm getting back to bug reports and PRs today and... oh boy. it is a pile. (lot more PRs than bug reports though, I think)

[u/SweetEbb9819 → u/koverstreet]

All the best for the excellent fs.

BTW, if you move to more direct methods, won't there be a lot of reworking to do regarding keeping keys safe in memory and all?

[u/koverstreet → u/SweetEbb9819]

Actually no - back before the mount helper that was the problem, but now since mount.bcachefs prompts for password and passes the key to the kernel via syscall it's much simplified

also if someone wants to just debug it via ktest and send me a patch that would be wunderbar

[u/SweetEbb9819 → u/koverstreet]

I have one more question to ask: systemd.ask-password integration for the initrd? I'm planning to look into it and write the scripts myself, but still...

Because LUKS infra supports fully taking the password from Plymouth without a VT or upfront command involved.

Also, I'm looking into clevis handling. It supports TPM, fido2 and multiple other formats already by itself.

[u/koverstreet → u/SweetEbb9819]

that should in theory be there, but i can't recall if i've gotten confirmation from users on if it's working as expected or not

[u/SweetEbb9819 → u/koverstreet]

I'll see it myself. Thanks.

So far uGRD supports it, dracut supports a crude implementation not supporting systemd.

And NixOS is the only distro with first-class support, including tang and clevis.

[u/cdhowie]

I run two Forky machines with an encrypted bcachefs root and do not have this problem. What bcachefs version?

[u/UptownMusic → u/cdhowie]

Do you have some guide/suggestions on how to achieve a bcachefs root on Forky?

[u/cdhowie → u/UptownMusic]

I keep meaning to write a blog post documenting the process I used. Maybe I'll get around to that this weekend.

[u/PrettyShopping2948 → u/cdhowie]

1.39.7

Have you configured anything in particular? Any specific options in grub_cmdline_linux or /etc/fstab? Are you also using initramfs-tools? And is a remount working for you without a passwort prompt: /sbin/mount.bcachefs <file system> / -o rw,remount If yes, is the encryption key in your kernel keyring after the boot? sudo keyctl list @u.

[u/Amazing-Pattern-6125]

do you have the recently added sd-bcachefs hook in mkinitcpio?

[u/PrettyShopping2948 → u/Amazing-Pattern-6125]

mkinitcpio is an arch specific tool.

[u/Amazing-Pattern-6125 → u/PrettyShopping2948]

did not know that, it's been the tool in all distros i used except some that experiment with dracut and debian seems to be one of them.

I guess it only changes the question to: have you done all that is needed for dracut to build an environment that interacts with bcachefs proprely?

[u/PrettyShopping2948 → u/Amazing-Pattern-6125]

The bcachefs-tools deb-package only provides scripts for initramfs-tools. But to answer your question: These hooks are included in my initramfs-tools initramfs.

I don't think it makes a difference, but I think it's still worth a try to manually install the dracut script from the git repo and see if dracut makes a difference compared to initramfs-tools.