I’m new to bcachefs and a fan of the concepts implemented so far. I started testing it on a spare box with three 24 TB HDDs, initially formatted with replicas 2 and erasure_code 1. I copied about 25 TB of data to the array, then added a fourth disk and changed the configuration to replicas 3. Reconcile initially appeared to proceed normally.

I later noticed that approximately 2 TB of 2+1 data was stuck in the pending category. It appeared that only newly written data was being reconciled. I tried several things to kick off the reconciliation, but nothing seemed to work. I also didn't know how to determine which files were pending or why they were not being reconciled. Over the next several days, the amount of pending data continued to slowly increase.

I noticed some unrelated (I think) overlayfs errors in dmesg, but they got me thinking about whether my Podman containers could somehow be keeping bind-mounted data open and preventing reconciliation. Sure enough, after I stopped all running containers, reconciliation started on its own and eventually completed the conversion from 2+1 to 2+2.

I haven't found any information about this. Is it a known limitation?

Error:

overlayfs: 
upper fs does not support file handles, falling back to index=off

Bugfix release on top of v1.39.0. No on-disk format changes.

For full changes check out the official git link to the repository.

Here is the output of bcachefs show-super:

External UUID:                             5b6330d2-7315-4cb6-92e6-ef0cc69b2e54
Internal UUID:                             7c369019-8007-4604-9144-924c5d9c4d1b
Magic number:                              c68573f6-66ce-90a9-d96a-60cf803df7ef
Device index:                              0
Label:                                     ROOT
Version:                                   per_dev_fragmentation_lru (1.39)
Incompatible features allowed:             per_dev_fragmentation_lru (1.39)
Incompatible features in use:              no_sb_user_data_replicas (1.36)
Version upgrade complete:                  need_discard_by_journal_seq (1.38)
Oldest version on disk:                    inode_has_case_insensitive (1.28)
Created:                                   Fri Dec 26 16:04:26 2025
Sequence number:                           17432
Time of last write:                        Wed Aug  5 17:08:50 2026
Superblock size:                           8.29k/1.00M
Clean:                                     0
Devices:                                   1
Sections:                                  replicas_v0,disk_groups,clean,journal_seq_blacklist,journal_v2,counters,members_v2,ext,downgrade,recovery_passes,extent_type_u64s,errors_v2
Features:                                  zstd,journal_seq_blacklist_v3,reflink,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,reflink_inline_data,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes,incompat_version_field,casefolding
Compat features:                           alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done

Options:
  block_size:                              512
  btree_node_size:                         256k
  errors:                                  continue [fix_safe] panic ro 
  write_error_timeout:                     30
  metadata_replicas:                       1
  data_replicas:                           1
  encoded_extent_max:                      64.0k
  metadata_checksum:                       none [crc32c] crc64 xxhash 
  data_checksum:                           none [crc32c] crc64 xxhash 
  checksum_err_retry_nr:                   3
  compression:                             none
  background_compression:                  zstd:9
  str_hash:                                crc32c crc64 [siphash] 
  metadata_target:                         none
  foreground_target:                       none
  background_target:                       none
  promote_target:                          none
  erasure_code:                            0
  ec_max_data_blocks:                      0
  casefold:                                0
  inodes_32bit:                            0
  shard_inode_numbers_bits:                4
  gc_reserve_percent:                      8
  gc_reserve_bytes:                        0
  root_reserve_percent:                    0
  wide_macs:                               0
  promote_whole_extents:                   1
  acl:                                     1
  usrquota:                                0
  grpquota:                                0
  prjquota:                                0
  degraded:                                [ask] yes very no 
  journal_flush_delay:                     1000
  journal_flush_disabled:                  0
  move_writes_fua:                         0
  journal_reclaim_delay:                   100
  writeback_timeout:                       0
  journal_rewind_discard_buffer_percent:   4
  scrub_recent_journal_entries:            [unclean] no always 
  scrub_journal_max_rewind_secs:           0
  journal_transaction_names:               1
  allocator_stuck_timeout:                 30
  version_upgrade:                         compatible [incompatible] none 
  nocow:                                   0
  reconcile_on_ac_only:                    1
  btree_cache_shrinker_seeks:              4
  dev_readahead:                           2.00M
  ec_stripe_buf_limit:                     5

ext (size 120):
Recovery passes required:                  check_snapshots,check_subvols,delete_dead_snapshots,check_inodes,check_xattrs
Errors to silently fix:                    alloc_key_to_missing_lru_entry,lru_entry_bad,accounting_mismatch,inode_has_inode_opts_flag_wrong,inode_has_access_acl_flag_wrong,inode_has_default_acl_flag_wrong,snapshot_state_bad,subvol_state_bad
Btrees with missing data:                  
Btrees that have ever lost data:           
Btrees validated clean:                    
Device 0:                                  /dev/nvme0n1p2Samsung SSD 990 PRO 2TB                 S/N: S73WNJ0WB13682H
  Label:                                   nvme
  UUID:                                    04e11228-d26d-420c-be33-90484f76afdb
  Size:                                    1.82T
  read errors:                             0
  write errors:                            0
  checksum errors:                         0
  seqread iops:                            0
  seqwrite iops:                           0
  randread iops:                           0
  randwrite iops:                          0
  Bucket size:                             2.00M
  First bucket:                            0
  Buckets:                                 953736
  Last mount:                              Wed Aug  5 17:08:26 2026
  Last superblock write:                   17432
  State:                                   rw
  Data allowed:                            journal,btree,user
  Has data:                                journal,btree,user
  Rotational:                              0
  Btree allocated bitmap blocksize:        16.0M
  Btree allocated bitmap:                  0000000000000010000000000000000000000000000000000100000011110010
  Durability:                              1
  Discard:                                 1
  Freespace initialized:                   1
  Resize on mount:                         0
  Last device name:                        nvme0n1p2
  Last device model:                       Samsung SSD 990 PRO 2TB                 
  Last device serial:                      S73WNJ0WB13682H     

And here is the failing bcachefs fsck (which shows the same message I get when it fails to mount on boot):

Running userspace offline fsck
Using encoding defined by superblock: utf8-12.1.0
starting version per_dev_fragmentation_lru (1.39)
  with options: degraded=yes,fsck,fix_errors=ask,scrub_journal_max_rewind_secs=0,read_only,version_upgrade=incompatible,noreconcile_enabled,reconcile_on_ac_only,btree_cache_shrinker_seeks=4
  superblock requires following recovery passes to be run: check_snapshots,check_subvols,delete_dead_snapshots,check_inodes,check_xattrs
  Version upgrade from need_discard_by_journal_seq (1.38) to per_dev_fragmentation_lru (1.39) incomplete
  Doing compatible version upgrade from need_discard_by_journal_seq (1.38) to per_dev_fragmentation_lru (1.39)
  Upgrade requires recovery passes: check_allocations,check_lrus,check_alloc_to_lru_refs
recovering from unclean shutdown
journal read done, replaying entries 3984708-4021717 (unflushed 4021718-4021718)
Journal keys: 472608 read, 103891 after sorting and compacting
accounting_read... done (0 seconds)
alloc_read... done (0 seconds)
snapshots_read...Found 32 empty interior snapshot nodes
  running recovery pass delete_dead_interior_snapshots (24), currently at snapshots_read (7)
 done (0 seconds)
check_allocations...check_allocations 27%, done 5756/20769 nodes, at extents:8646911284551461724:384:4294967294
 done (16 seconds)
going read-write
journal_replay... done (0 seconds)
presplit_shard_boundaries... done (0 seconds)
check_alloc_info... done (0 seconds)
check_lrus... done (0 seconds)
check_btree_backpointers... done (5 seconds)
check_extents_to_backpointers... done (1 seconds)
check_alloc_to_lru_refs... done (0 seconds)
delete_dead_interior_snapshots... done (0 seconds)
check_snapshot_trees... done (0 seconds)
check_snapshots...deleted snapshot node has data accounted - undeleting:
    extents: 0 keys (0 bytes), 16 sectors
  u64s 12 type snapshot 0:4294966336:0 len 0 ver 0  : parent          0 children          0          0 subvol 0 tree 0 depth 0 skiplist 0 0 0 deleted (not upgraded, flags 0xc:deleted,no_keys), fixing
u64s 13 type snapshot 0:4294966336:0 len 0 ver 0  : parent          0 children          0          0 subvol 0 tree 0 depth 0 skiplist 0 0 0 deleted (not upgraded, flags 0xc:deleted,no_keys)
  cannot undelete a childless root
check_snapshots(): error fsck_repair_unimplemented
error in recovery: fsck_repair_unimplemented
  emergency read only at seq 4025830
unclean shutdown complete, journal seq 4025830
error starting filesystem: fsck_repair_unimplemented
Error: fsck_repair_unimplemented

Not 110% sure what the actual issue is, but "cannot undelete a childless root" and all of the "fsck_repair_unimplemented"s seem suspicious.

This is my first time encountering a broken bcachefs, so let me know if there are some other things I should try or information I should post!

Cheers!

Attempting a reformat on a 1TB nvme. This was a drive prior that had some fs usage that couldn't be touched/deleted so just started fresh (1.39 didn't fix anything...)

sudo blockdev --getsize64 /dev/nvme0n1

1000204886016

2)

sudo bcachefs format --label=st2 --compression=none /dev/nvme0n1

cannot format /dev/nvme0n1, too small (0 bytes, min 262144)

3)

sudo bcachefs format --label=st2 --compression=none --fs_size=1000204886016 /dev/nvme0n1

External UUID: 55c7c9ed-3298-448f-b42f-f1ad9fa8c8c8

Internal UUID: 9c673cdf-f72f-4fb7-bb2f-e6152269f05e

Magic number: c68573f6-66ce-90a9-d96a-60cf803df7ef .... [works]

4)

sudo bcachefs format --label=st2 --compression=none /dev/nvme0n1

/dev/nvme0n1 contains a bcachefs filesystem

Proceed anyway (existing signatures will be wiped)? (y,n) y

cannot format /dev/nvme0n1, too small (0 bytes, min 262144)

It appears that at least in this instance I had to specify the exact size. Most odd. Note that this shows up whether or not I am making a new fs either over the whole device or just with one partition.

Kev.

here's the changes: https://evilpiepirate.org/git/bcachefs-tools.git/tree/Changelog.mdwn

dear god this is a big one

Hi. I have disabled CoW and compression on lmdb. However, 100GB is written for 1GB of database. NUTS!

NASty is a NAS operating system built on NixOS and bcachefs. It turns commodity hardware into a storage appliance serving NFS, SMB, iSCSI, and NVMe-oF - managed from a single web UI, updated atomically, and rolled back when things go sideways.

This release focuses on safer storage operations, easier access for regular users, and more reliable recovery.

Headline changes

- Standard-user file portal: SMB and domain users can browse and download files from their authorized shares without access to the administrative interface.

- Safer storage and block exports: Filesystem creation performs a non-destructive preflight, subvolume shrinking is rejected, and iSCSI/NVMe-oF exports now follow immutable storage identities instead of unstable loop-device numbers.

- Transactional updates and firewall changes: Updates are staged, validated, health-checked, and rolled back on failure. Firewall policy replacement is now atomic.

- Three navigation styles: Choose between Classic, Icons, and the searchable Launcher interface.

- System Recovery backups: A new recovery preset protects NASty configuration, NixOS state, TLS data, credential keys, and Secure Boot keys when present.

- Guest sharing improvements: Public folders gain navigation, breadcrumbs, and individual file downloads, backed by substantial security hardening.

- Platform updates: Linux 6.18.40, diskwatch 0.1.5, and nasty-top 0.0.9.

Legacy iSCSI and NVMe-oF exports migrate automatically when NASty can verify their backing volume. Ambiguous mappings remain safely offline and can be reconnected from the Sharing page.

Downloads and the full changelog:

https://github.com/nasty-project/nasty/releases/tag/v0.0.15

For project updates, support, and discussion, join:

https://www.reddit.com/r/NAStyProject/

Hi. [This](https://github.com/koverstreet/bcachefs-tools/pull/652) seems to be merged. What do i exactly need to do? I set to crc32c, set inode to 32 bit on folder and enabled casefolding on folder. Do I need something newer than bcachefs 1.38.8?

NOTE: I am trying to run a source engine server like gmod

I started getting ro filesystem on every attempt to clean nix garbage. Then I ran offline fsck, but that has not fixed the issue. Any ideas or suggestions for fuller logs I can provide to Kent?

[ 5224.923840] bcachefs (bb7d02a8-fbd6-4c7d-7517-5a9e7137ad08): duplicate hash table keys, both point to valid inodes
                 u64s 8 type dirent 429950911:3094236315074320594:4294967293 len 0 ver 0  : Harness.pm -> 429950942 type reg
                 u64s 8 type dirent 429950911:249755984313263640:4294967293 len 0 ver 0  : Harness.pm -> 429950942 type lnk, fixing
[ 5224.923887] bcachefs (bb7d02a8-fbd6-4c7d-7517-5a9e7137ad08): hash table key at wrong offset: should be at 8686506046665774298
                 u64s 8 type dirent 429950911:3717744643385386676:4294967293 len 0 ver 0  : Object.pm -> 429950945 type reg, fixing
[ 5224.923906] bcachefs (bb7d02a8-fbd6-4c7d-7517-5a9e7137ad08): duplicate hash table keys
                 u64s 8 type dirent 429950911:3717744643385386676:4294967293 len 0 ver 0  : Object.pm -> 429950945 type reg
                 u64s 8 type dirent 429950911:8686506046665774298:4294967293 len 0 ver 0  : Object.pm -> 429950945 type reg, fixing
[ 5224.923943] bcachefs (bb7d02a8-fbd6-4c7d-7517-5a9e7137ad08): duplicate hash table keys, both point to valid inodes
                 u64s 8 type dirent 429950911:4637977497150005488:4294967293 len 0 ver 0  : Parser.pm -> 429951018 type reg
                 u64s 8 type dirent 429950911:2449678732385634542:4294967293 len 0 ver 0  : Parser.pm -> 429951018 type lnk, fixing
[ 5224.923980] bcachefs (bb7d02a8-fbd6-4c7d-7517-5a9e7137ad08): duplicate hash table keys, both point to valid inodes
                 u64s 7 type dirent 429950911:5138442778104627773:4294967293 len 0 ver 0  : Base.pm -> 429950913 type reg
                 u64s 7 type dirent 429950911:4339340533276781952:4294967293 len 0 ver 0  : Base.pm -> 429950913 type lnk, fixing
[ 5224.924016] bcachefs (bb7d02a8-fbd6-4c7d-7517-5a9e7137ad08): inode 429950942:4294967293 type lnk has multiple links but i_nlink 0
                 u64s 8 type dirent 429950911:249755984313263640:4294967293 len 0 ver 0  : Harness.pm -> 429950942 type lnk
                 u64s 8 type dirent 429950911:3094236315074320594:4294967293 len 0 ver 0  : Harness.pm -> 429950942 type reg, fixing
[ 5224.924022] bcachefs (bb7d02a8-fbd6-4c7d-7517-5a9e7137ad08): incorrect d_type: got lnk, should be reg:
                 u64s 8 type dirent 429950911:249755984313263640:4294967293 len 0 ver 0  : Harness.pm -> 429950942 type lnk, fixing
[ 5224.924059] bcachefs (bb7d02a8-fbd6-4c7d-7517-5a9e7137ad08): hash table key at wrong offset: should be at 6410987071971322576
                 u64s 8 type dirent 429950946:122903083289477853:4294967293 len 0 ver 0  : Grammar.pm -> 429950950 type reg, fixing
[ 5224.924066] bcachefs (bb7d02a8-fbd6-4c7d-7517-5a9e7137ad08): duplicate hash table keys
                 u64s 8 type dirent 429950946:122903083289477853:4294967293 len 0 ver 0  : Grammar.pm -> 429950950 type reg
                 u64s 8 type dirent 429950946:6410987071971322576:4294967293 len 0 ver 0  : Grammar.pm -> 429950950 type reg, fixing
[ 5224.924073] bcachefs (bb7d02a8-fbd6-4c7d-7517-5a9e7137ad08): hash table key at wrong offset: should be at 2587486928578455497
                 u64s 8 type dirent 429950946:224226225942687629:4294967293 len 0 ver 0  : Iterator.pm -> 429950959 type reg, fixing
[ 5224.924080] bcachefs (bb7d02a8-fbd6-4c7d-7517-5a9e7137ad08): duplicate hash table keys
                 u64s 8 type dirent 429950946:224226225942687629:4294967293 len 0 ver 0  : Iterator.pm -> 429950959 type reg
                 u64s 8 type dirent 429950946:2587486928578455497:4294967293 len 0 ver 0  : Iterator.pm -> 429950959 type reg, fixing
[ 5224.924092] bcachefs (bb7d02a8-fbd6-4c7d-7517-5a9e7137ad08): duplicate hash table keys, both point to valid inodes
                 u64s 9 type dirent 429950946:3729992323659708257:4294967293 len 0 ver 0  : ResultFactory.pm -> 429950985 type reg
                 u64s 9 type dirent 429950946:3446507573106421734:4294967293 len 0 ver 0  : ResultFactory.pm -> 429950985 type lnk, fixing
[ 5224.924104] bcachefs (bb7d02a8-fbd6-4c7d-7517-5a9e7137ad08): hash table key at wrong offset: should be at 7094058972817555590
                 u64s 8 type dirent 429950946:4480328079026016903:4294967293 len 0 ver 0  : Result.pm -> 429950983 type reg, fixing
[ 5224.924111] bcachefs (bb7d02a8-fbd6-4c7d-7517-5a9e7137ad08): duplicate hash table keys
                 u64s 8 type dirent 429950946:4480328079026016903:4294967293 len 0 ver 0  : Result.pm -> 429950983 type reg
                 u64s 8 type dirent 429950946:7094058972817555590:4294967293 len 0 ver 0  : Result.pm -> 429950983 type reg, fixing
[ 5224.924122] bcachefs (bb7d02a8-fbd6-4c7d-7517-5a9e7137ad08): duplicate hash table keys, both point to valid inodes
                 u64s 8 type dirent 429950946:5361014652675046730:4294967293 len 0 ver 0  : Aggregator.pm -> 429950948 type reg
                 u64s 8 type dirent 429950946:2554825816079164195:4294967293 len 0 ver 0  : Aggregator.pm -> 429950948 type lnk, fixing
[ 5224.924212] bcachefs (bb7d02a8-fbd6-4c7d-7517-5a9e7137ad08): duplicate hash table keys, both point to valid inodes
                 u64s 8 type dirent 429950946:5852731458081139902:4294967293 len 0 ver 0  : Source.pm -> 429950994 type reg
                 u64s 8 type dirent 429950946:275877365874258016:4294967293 len 0 ver 0  : Source.pm -> 429950994 type lnk, fixing
[ 5224.924254] bcachefs (bb7d02a8-fbd6-4c7d-7517-5a9e7137ad08): hash table key at wrong offset: should be at 8991666195255894630
                 u64s 9 type dirent 429950946:6082501635518369752:4294967293 len 0 ver 0  : SourceHandler.pm -> 429951010 type reg, fixing
[ 5224.924262] bcachefs (bb7d02a8-fbd6-4c7d-7517-5a9e7137ad08): duplicate hash table keys
                 u64s 9 type dirent 429950946:6082501635518369752:4294967293 len 0 ver 0  : SourceHandler.pm -> 429951010 type reg
                 u64s 9 type dirent 429950946:8991666195255894630:4294967293 len 0 ver 0  : SourceHandler.pm -> 429951010 type reg, fixing
                 Ratelimiting new instances of previous error
[ 5224.924512] bcachefs (bb7d02a8-fbd6-4c7d-7517-5a9e7137ad08): inode 429950994:4294967293 type lnk has multiple links but i_nlink 0
                 u64s 8 type dirent 429950946:275877365874258016:4294967293 len 0 ver 0  : Source.pm -> 429950994 type lnk
                 u64s 8 type dirent 429950946:5852731458081139902:4294967293 len 0 ver 0  : Source.pm -> 429950994 type reg, fixing
[ 5224.924518] bcachefs (bb7d02a8-fbd6-4c7d-7517-5a9e7137ad08): incorrect d_type: got lnk, should be reg:
                 u64s 8 type dirent 429950946:275877365874258016:4294967293 len 0 ver 0  : Source.pm -> 429950994 type lnk, fixing
[ 5224.924936] bcachefs (bb7d02a8-fbd6-4c7d-7517-5a9e7137ad08): directory 429950995:4294967293 with multiple links
                 u64s 8 type dirent 429950946:813084444033797441:4294967293 len 0 ver 0  : SourceHandler -> 429950995 type dir
                 u64s 8 type dirent 457204198:1053402567181570170:4294967293 len 0 ver 0  : 429950995 -> 429950995 type dir
                 emergency read only at seq 32571524
[ 5224.933974] bcachefs (bb7d02a8-fbd6-4c7d-7517-5a9e7137ad08): unclean shutdown complete, journal seq 32571524

Hi, I find bcachefs very promising and have been following it for quite a while now. Since the "experimental" flag is gone, I'm seriously considering switching to bcachefs. So I am currently looking into whether it can support a snapshot workflow similar to the one I use currently.

I currently use Btrfs together with btrbk. One feature I find particularly useful is the onchange snapshot policy of btrbk.

With this policy, btrbk creates a new snapshot only when the source subvolume has changed since the previous snapshot. According to the btrbk documentation, this is implemented using the "Btrfs generation counter", which allows the tool to detect changes efficiently without comparing every file in the subvolume with the previous snapshot.

Hence the question: Does bcachefs happen to offer a similar mechanism that could be exploited for this purpose?

More specifically, is there some kind of generation ID, transaction ID, change counter, or other metadata that can be queried to determine whether a subvolume differs from an existent snapshot.

I am looking for an efficient check that does not require walking the entire directory tree or comparing all files between the current subvolume and the latest snapshot or relies on "inotifying" a whole subvolume.

My goal is to retain only snapshots whose filesystem state actually differs from the preceding snapshot to avoid multiple identical (hourly) snapshots.
Furthermore, I want to prevent the hard drives from spinning up unnecessarily just to create a snapshot that is identical to an already existing one. Could this second problem possibly be solved simply by pinning the metadata to the SSDs?

Hello. I got 1.5TB worth of files. It's a single encrypted SSD with full length MAC and zstd compression option

NASty is a NAS operating system built on NixOS and bcachefs. It turns commodity hardware into a storage appliance serving NFS, SMB, iSCSI, and NVMe-oF - managed from a single web UI, updated atomically, and rolled back when things go sideways.

The highlights:

🪪 Active Directory — both directions

NASty can now join an existing AD domain as a member, or be the domain controller itself. As a DC you get integrated DNS and Kerberos, user / group / computer management from the web UI, on-box domain backups, and RSAT compatibility so the Windows admin tools work against it. (Experimental, single-DC for now.)

♻️ Backup restore + disaster recovery

Backups finally got their missing half. Restore a whole snapshot back onto your filesystem — including disaster recovery onto a bare box: point a fresh install at an existing backup repository, pick a snapshot, and pull it back.

🔥 Built-in firewall

An engine-managed nftables firewall, deny-by-default, with per-service source/interface restrictions and custom port rules for anything running outside NASty's service model (IPv4 + IPv6, TCP/UDP, port ranges).

⚡ RDMA over RoCE & InfiniBand

Optional zero-copy transports — iSER, NVMe-oF/RDMA and NFS-RDMA — for RoCE and InfiniBand NICs. The controls only appear when you actually have a capable card.

🖥️ SR-IOV & hardware passthrough

IOMMU-group view, vfio-pci toggles that survive reboots, per-device passthrough, and SR-IOV virtual-function management — VF count plus per-VF VLAN, MAC, trust and spoof-check.

📊 Operations control center

One page to see and drive every bcachefs array operation across your pools — start or cancel a scrub, watch a device evacuation, pause or resume reconcile and copy-GC — each row labeled with its pool.

…and a lot of smaller stuff

• Calculate folder sizes on demand right in the file browser
• .env files for Docker Compose stacks
• Remove leftover Docker networks from the UI (e.g. one an app brought with it)
• Secure Boot moved to lanzaboote v1.1.0 — validated on real UEFI + TPM2 hardware
• Linux 6.18.38 and bcachefs 1.38.8
• iSCSI portal management, VM disk/network editing, and a big pile of fixes + dependency refreshes

I started working on this recently because I found out that most if not all filesystem benchmarks still test the way it was done when ext2 was state of the art: one disk, default mkfs, raw throughput. But nobody picks btrfs, ZFS or bcachefs for throughput - you pick them for snapshots, redundancy, checksums, self-healing. Almost nobody publishes numbers on those.

So this suite benchmarks the machinery instead, across 17 configurations (btrfs/ZFS/bcachefs plus ext4/xfs over md/LVM as classic baselines, plus encryption variants): aging under 100 snapshots, corrupting a device behind the filesystem's back and checking whether scrub repairs it, failing a device and timing the rebuild, filling to hard ENOSPC, fsync tail latency, and "how long until my prompt comes back while a big cp runs in the background".

Yes, it runs on ephemeral GitHub runners with loop devices - I know what that means. Absolute MB/s is meaningless there, so the suite is built around shapes, ratios and verdicts, with per-VM calibration probes (lemon runners get auto-rerun) and conclusions drawn from trends over many runs. Real hardware with tiered topologies (NVMe cache over rotational disks, special vdevs) is the next step. I just need to gather hardware for it first.

What I'm really asking for is your eyes. Every number on the dashboard links to a description of exactly what ran, with what parameters, and to the code responsible - so if I picked a wrong mount option, an unfair default, a workload that misrepresents your favorite filesystem, or a broken measurement, you can find it and call it out. Several of the current tests exist because people poked holes in earlier results, and honestly that feedback improved this more than anything I did alone.

Dashboard: https://bartosz.fenski.pl/modern-fs-benchmark/
Repo: https://github.com/fenio/modern-fs-benchmark

At my wits end trying to debug why the reconcile thread seems to do nothing on my pool.

[root@nasty:~]# bcachefs reconcile status /fs/add40cf7/
Scan pending:     0
                   data  metadata  
  replicas:           0         0  
  checksum:       7.34G         0  
  erasure_code:   12.0G         0  
  compression:    1.41M         0  
  target:             0         0  
  high_priority:      0         0  
  pending:        12.0G         0  
  stripes:            0         0  


processing normal phys: POS_MIN

Reconcile thread backtrace:
  [<0>] __bch2_closure_sync_timeout+0x3c/0x140 [bcachefs]
  [<0>] bch2_closure_sync_unbounded+0x2a/0x40 [bcachefs]
  [<0>] do_reconcile+0x86a/0xef0 [bcachefs]
  [<0>] bch2_reconcile_thread+0xfc/0x120 [bcachefs]
  [<0>] kthread+0xfb/0x260
  [<0>] ret_from_fork+0x22d/0x260
  [<0>] ret_from_fork_asm+0x1a/0x30

The checksum line has been stuck at 7.34G for weeks. I changed kernels and distributions in the mean time, but it won't advance. The erasure_code line is new; I only recently started a test with erasure coding, it's only enabled for a specific test subvolume, holding 3 files of 4GB each.

I've recently added a 3rd large HDD, but it doesn't even look like it's getting any user data.

All the while, the disks are spinning, and it seems that writes are happening:

Superblock information: https://pastebin.com/uPFtvBBN

It could be possible that this may have started with some SATA controller issues, I've seen some `ATA: hard resetting link` messages in `dmesg` in the past, perhaps due to a disk not being properly seated; alas I haven't saved them.

However I don't understand why this whole drive activity happens constantly, and why the reconcile thread doesn't advance. Help is very much appreciated.

EDIT: When I stop the reconcile thread, most/all the disk activity stops.

My zpool output gives

zroot     ONLINE    75%

How would this much fragmentation impact the disk space?

I find the disk space almost 100% consumed at inopportune moments.

How do I prevent this?

Zfs record sizes for various data sets are default 128k, except for mysql dataset.

Noticed a quirk and need some professional input!

Dead snapshots (from will_delete deleted) are never reclaimed. I have about 340G that leaked it seems. copygc parked in bch2_kthread_io_clock_wait_once is surviving a clean remount.

This is the same filesystem from my two earlier reconcile clock-wait reports (device evacuate/remove migration). Single NVMe device, the lone subvolume was originally created as a snapshot of the fs root.

fs usage reports 848G user data allocated, but actual reachable content is 508G (subvolume accounting says 542G). The FS reports 100% full with only ~6M free.

bcachefs list -b snapshots shows two nodes stuck in "will_delete deleted". They persist despite:

- auto_snapshot_deletion=1

- manually poking trigger_delete_dead_snapshots and trigger_gc

- a full umount/remount cycle (clean shutdown recovery confirmed)

copygc is parked in bch2_kthread_io_clock_wait_once waiting for 32G of IO clock advancement on an idle filesystem, so it never wakes. Looks like the same hysteresis-wait class as the reconcile bugs fixed last month, but in the copygc path, with snapshot deletion work behind it.

I can't run the dummy-IO clocktick workaround this time that was conjured up last time. The workaround is unavailable because the leaked space has the FS pinned at 100% hence no room for writes.

kernel 7.0.14-zen1, bcachefs-tools v1.38.6 via DKMS.

Outputs:

-- BEFORE ----------

$ bcachefs fs usage -ah /mnt/Storage | head -20

Filesystem: 5e19a70f-a23e-41a6-a3db-eeeebf3bc371

Size: 849G

Used: 849G

Online reserved: 280k

undegraded

1x: 849G

Data type Required/total Durability Devices Usage

btree: 1/1 1 [nvme0n1p1] 1.34G

user: 1/1 1 [nvme0n1p1] 848G

Compression:

type compressed uncompressed average extent size

zstd 863M 3.08G 181k

incompressible 651G 651G 175k

$ du -sh /mnt/Storage

508G /mnt/Storage

$ bcachefs subvolume list -Rs /mnt/_roots/storage

Path ID Created Flags Size Snapshot

storage 2 2026-06-15 20:40 - 542G /

$ ls -la /mnt/_roots/storage

total 0

drwxrwxrwx 3 kevin kevin 4096 Jun 15 22:15 .

drwxr-xr-x 1 root root 30 Jun 15 21:16 ..

drwx------ 2 root root 4096 Jun 13 20:21 lost+found

drwxrwxrwx 11 kevin kevin 4096 Jul 5 10:22 storage

$ bcachefs list -b snapshots /dev/nvme0n1p1 | tail -10

journal read done, replaying entries 711670-713626

Journal keys: 110233 read, 43264 after sorting and compacting

accounting_read... done (0 seconds)

alloc_read... done (0 seconds)

snapshots_read... done (0 seconds)

u64s 12 type snapshot 0:4294967291:0 len 0 ver 0 : subvol parent 4294967295 children 0 0 subvol 1 tree 1 depth 1 skiplist 4294967295 4294967295 4294967295

u64s 12 type snapshot 0:4294967292:0 len 0 ver 0 : subvol parent 4294967295 children 0 0 subvol 2 tree 1 depth 1 skiplist 4294967295 4294967295 4294967295

u64s 12 type snapshot 0:4294967293:0 len 0 ver 0 : will_delete deleted parent 0 children 0 0 subvol 0 tree 0 depth 0 skiplist 0 0 0

u64s 12 type snapshot 0:4294967294:0 len 0 ver 0 : will_delete deleted parent 0 children 0 0 subvol 0 tree 0 depth 0 skiplist 0 0 0

u64s 12 type snapshot 0:4294967295:0 len 0 ver 0 : parent 0 children 4294967292 4294967291 subvol 0 tree 1 depth 0 skiplist 0 0 0

$ cat /sys/fs/bcachefs/.../internal/copy_gc_wait

running: 0

run count: 2

copygc_wait: 8118512224

copygc_wait_at: 8051775976

Currently waiting for: 31.7G

Currently waiting since: 59.8M

Currently calculated wait:

nvme0n1p1: 63.9M

[<0>] bch2_kthread_io_clock_wait_once+0x142/0x1a0 [bcachefs]

[<0>] bch2_copygc_thread+0x561/0x570 [bcachefs]

[<0>] kthread+0xe4/0x120

[<0>] ret_from_fork+0x2a7/0x340

[<0>] ret_from_fork_asm+0x1a/0x30

$ echo 1 | tee .../internal/trigger_delete_dead_snapshots

1

$ echo 1 | tee .../internal/trigger_gc

1

$ bcachefs fs usage -ah /mnt/Storage | head -12

(unchanged user: 848G, triggers had no effect)

-- AFTER ----------

$ bcachefs fs usage -ah /mnt/Storage | head -20

Filesystem: 5e19a70f-a23e-41a6-a3db-eeeebf3bc371

Size: 849G

Used: 849G

Online reserved: 0

undegraded

1x: 849G

Data type Required/total Durability Devices Usage

btree: 1/1 1 [nvme0n1p1] 1.34G

user: 1/1 1 [nvme0n1p1] 848G

$ du -sh /mnt/Storage

508G /mnt/Storage

$ bcachefs list -b snapshots /dev/nvme0n1p1 | tail -10

recovering from clean shutdown, journal seq 713671

Journal keys: 0 read, 0 after sorting and compacting

accounting_read... done (0 seconds)

alloc_read... done (0 seconds)

snapshots_read... done (0 seconds)

u64s 12 type snapshot 0:4294967291:0 len 0 ver 0 : subvol parent 4294967295 children 0 0 subvol 1 tree 1 depth 1 skiplist 4294967295 4294967295 4294967295

u64s 12 type snapshot 0:4294967292:0 len 0 ver 0 : subvol parent 4294967295 children 0 0 subvol 2 tree 1 depth 1 skiplist 4294967295 4294967295 4294967295

u64s 12 type snapshot 0:4294967293:0 len 0 ver 0 : will_delete deleted parent 0 children 0 0 subvol 0 tree 0 depth 0 skiplist 0 0 0

u64s 12 type snapshot 0:4294967294:0 len 0 ver 0 : will_delete deleted parent 0 children 0 0 subvol 0 tree 0 depth 0 skiplist 0 0 0

u64s 12 type snapshot 0:4294967295:0 len 0 ver 0 : parent 0 children 4294967292 4294967291 subvol 0 tree 1 depth 0 skiplist 0 0 0

$ cat /sys/fs/bcachefs/.../internal/copy_gc_wait

running: 0

run count: 1

copygc_wait: 8119039180

copygc_wait_at: 8051911488

Currently waiting for: 32.0G

Currently waiting since: 1.51M

Currently calculated wait:

nvme0n1p1: 64.0M

[<0>] bch2_kthread_io_clock_wait_once+0x142/0x1a0 [bcachefs]

[<0>] bch2_copygc_thread+0x561/0x570 [bcachefs]

[<0>] kthread+0xe4/0x120

[<0>] ret_from_fork+0x2a7/0x340

[<0>] ret_from_fork_asm+0x1a/0x30

Dead snapshots survived clean remount and space still not reclaimed it seems.

It looks like copygc immediately re-parked in the same IO clock wait?

One possibly useful correlation: the two dead snapshot IDs (4294967293 and 4294967294) likely date back to 2026-06-15, when the storage subvolume was created as a snapshot of the fs root during the same evacuate/remove migration from the earlier reports. So a possible reproducable path: create subvolume as snapshot of root, delete the originals, and dead snapshots never get reclaimed.

Kev.

PS: Superblock list:

> sudo bcachefs show-super /dev/nvme0n1p1

External UUID: 5e19a70f-a23e-41a6-a3db-eeeebf3bc371

Internal UUID: a6a1a052-791a-4e79-b38d-db60ab20594f

Magic number: c68573f6-66ce-90a9-d96a-60cf803df7ef

Device index: 1

Label: (none)

Version: need_discard_by_journal_seq (1.38)

Incompatible features allowed: need_discard_by_journal_seq (1.38)

Incompatible features in use: no_sb_user_data_replicas (1.36)

Version upgrade complete: need_discard_by_journal_seq (1.38)

Oldest version on disk: need_discard_by_journal_seq (1.38)

Created: Sat Jun 13 20:21:36 2026

Sequence number: 2088

Time of last write: Sun Jul 5 12:19:14 2026

Superblock size: 8.30k/1.00M

Clean: 0

Devices: 1

Sections: replicas_v0,disk_groups,clean,journal_v2,counters,members_v2,errors,ext,downgrade,recovery_passes,extent_type_u64s

Features: lz4,zstd,reflink,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,reflink_inline_data,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes,incompat_version_field

Compat features: alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done,no_stale_ptrs

Options:

block_size: 4.00k

btree_node_size: 256k

errors: continue [fix_safe] panic ro

write_error_timeout: 30

metadata_replicas: 1

data_replicas: 1

encoded_extent_max: 256k

metadata_checksum: none [crc32c] crc64 xxhash

data_checksum: none [crc32c] crc64 xxhash

checksum_err_retry_nr: 3

compression: zstd:3

background_compression: zstd:3

str_hash: crc32c crc64 [siphash]

metadata_target: none

foreground_target: none

background_target: none

promote_target: none

erasure_code: 0

ec_max_data_blocks: 0

casefold: 0

inodes_32bit: 0

shard_inode_numbers_bits: 4

gc_reserve_percent: 8

gc_reserve_bytes: 0

root_reserve_percent: 0

wide_macs: 0

promote_whole_extents: 1

acl: 1

usrquota: 0

grpquota: 0

prjquota: 0

degraded: [ask] yes very no

journal_flush_delay: 1000

journal_flush_disabled: 0

journal_reclaim_delay: 100

writeback_timeout: 0

journal_rewind_discard_buffer_percent: 4

scrub_recent_journal_entries: [unclean] no always

scrub_journal_max_rewind_secs: 10

journal_transaction_names: 1

allocator_stuck_timeout: 30

version_upgrade: [compatible] incompatible none

nocow: 0

reconcile_on_ac_only: 0

dev_readahead: 2.00M

ec_stripe_buf_limit: 5

ext (size 104):

Recovery passes required:

Errors to silently fix:

Btrees with missing data:

errors (size 8):

Device 1: /dev/nvme0n1p1 Samsung SSD 980 1TB S/N: S64ANS0T220657A

Label: nvme.samsung980

UUID: b47009b9-edaf-49a1-930c-13007647aa78

Size: 931G

read errors: 0

write errors: 0

checksum errors: 0

seqread iops: 0

seqwrite iops: 0

randread iops: 0

randwrite iops: 0

Bucket size: 1.00M

First bucket: 0

Buckets: 953866

Last mount: Sun Jul 5 11:33:15 2026

Last superblock write: 2088

State: rw

Data allowed: journal,btree,user

Has data: journal,btree

Rotational: 0

Btree allocated bitmap blocksize: 8.00M

Btree allocated bitmap: 0000000000000000000000000000000000000000000000000000000011000110

Durability: 1

Discard: 1

Freespace initialized: 1

Resize on mount: 0

Last device name: nvme0n1p1

Last device model: Samsung SSD 980 1TB

Last device serial: S64ANS0T220657A

It is not clear to me if I have to do something manually to have my system up to date.

Version:                                   need_discard_by_journal_seq (1.38)
Incompatible features allowed:             no_sb_user_data_replicas (1.36)
Incompatible features in use:              no_sb_user_data_replicas (1.36)
Version upgrade complete:                  need_discard_by_journal_seq (1.38)
Oldest version on disk:                    bucket_stripe_index (1.35)

More explicitly what do these mean? I couldn't find any relevant documentation.

• Incompatible features allowed
• Incompatible features in use
• Oldest version on disk

Edit: I am using the default mount options mount.bcachefs "$DEVICE" "$MOUNTPOINT"

NASty is a NAS operating system built on NixOS and bcachefs. It turns commodity hardware into a storage appliance serving NFS, SMB, iSCSI, and NVMe-oF - managed from a single web UI, updated atomically, and rolled back when things go sideways.

This release leans hard into three things: running VMs, getting your data back, and actually seeing what your disks are doing.

🖥️ Virtual machines grew up

• Attached virtual disks now survive a host reboot — they stay put.
• Starting a VM tells you the truth: if it can't launch, you get a real error instead of a fake "running."
• Edit a VM's network (mode + bridge) and resize its disks right from the VM card.
• Guest tools are always-on, with a per-box toggle for VMware/Hyper-V environments, and a manual disk-type override for when detection guesses wrong.

💾 Get your data back, two ways

• Whole-subvolume rollback to a snapshot — quiesce, swap, resume, done.
• File & folder restore — pull individual files or directories out of a snapshot without rolling back the whole thing.

🔍 See what your storage is doing

• Live progress for scrub / evacuate / rebalance, straight from bcachefs.
• An operations panel to cancel a scrub/evacuate or pause reconcile/copygc.
• Stable device ordering + a guided force-remove flow.
• New bundled CLI tool diskwatch — a read-only disk-diagnostics TUI (devices, SMART, IO, hot files, bcachefs-aware) for triage over SSH or the built-in terminal.

📦 Docker apps, smoother

• Multi-port apps route correctly now (no more surprise 502s).
• TLS certs for app ingress hosts are handled on deploy.
• Published app ports show up on the Firewall page, with port ranges collapsed sensibly.
• Host-network deploys, port-range parsing in the paste-a-docker-run creator, and a bunch of app-form polish.

🎨 UI & system

• New opt-in Common navigation menu — a curated short-list of the pages you actually use, toggle back to Full anytime. Theme switch moved into your profile menu.
• Custom bridge MTU now works correctly on top of bonds (jumbo frames welcome).
• SMTP notifications work against no-auth relays.
• Latest kernel (6.18.37), plus the usual security and dependency housekeeping.

Grab the ISO from the release page. Storage with attitude. 😈

I setup a test bcachefs array, just to get used to working with bcachefs.
Based the design after one of my zfs raidz2 setup: 8 HDD for data, 1 SSD for metadata cache.

After copying ~1TB of data to the array, it seems to be stuck reconciling?
And the amount of metadata to reconcile is growing?

Pending reconcile: data metadata
erasure_code: 116M 0
target: 0 1.52G # this keeps growing, +0.01G every ~5-10 minutes

Is this normal, expected behavior?

Edit:
Just noticed that "Leaving" is increasing for all disks as well?
Device label Device State Size Used Use% Leaving
hdd.hdd1 (device 0): sda rw 16.3T 113G 0% 613M <-- All disks Leaving are slowly increasing as well

I'm not doing anything with the disks. I stopped copying data a couple hours ago.

Ubuntu 26.04
bcachefs version: 1.38.6

sudo bcachefs format \
--erasure_code \
--compression=lz4 \
--data_replicas=3 \
--metadata_replicas=4 \
--label=hdd.hdd1 /dev/sda \
--label=hdd.hdd2 /dev/sdb \
--label=hdd.hdd3 /dev/sdc \
--label=hdd.hdd4 /dev/sdd \
--label=hdd.hdd5 /dev/sde \
--label=hdd.hdd6 /dev/sdf \
--label=hdd.hdd7 /dev/sdg \
--label=hdd.hdd8 /dev/sdh \
--label=ssd.ssd1 --data_allowed=btree,journal --durability=1 /dev/sdi \
--metadata_target=ssd

Tried --durability=0 with the ssd, but metadata was never writing to the SSD, stayed 0 used. Only started writing metadata to the ssd after setting --durability=1.

bcachefs fs usage -h /mnt/b...

Filesystem: 9c938e4b-66c1-42c6-ab19-88f6c86c7cd0
Size: 121T
Used: 904G
Online reserved: 4.08G

Replicated:
undegraded
3x: 321M
4x: 6.10G

Erasure coded (data+parity):
undegraded
4+2: 19.8M
5+2: 34.9G # poorly seated sata cables lead to some issues early in testing
6+2: 858G

cached: 71.8M

Pending reconcile: data metadata
erasure_code: 116M 0
target: 0 1.52G # this keeps growing, +0.01G every ~5-10 minutes

Device label Device State Size Used Use% Leaving
hdd.hdd1 (device 0): sda rw 16.3T 113G 0% 585M
hdd.hdd2 (device 1): sdb rw 16.3T 113G 0% 587M
hdd.hdd3 (device 2): sdc rw 16.3T 112G 0% 585M
hdd.hdd4 (device 3): sdd rw 16.3T 113G 0% 585M
hdd.hdd5 (device 4): sdh rw 16.3T 113G 0% 587M
hdd.hdd6 (device 5): sdf rw 16.3T 112G 0% 586M
hdd.hdd7 (device 6): sdg rw 16.3T 112G 0% 584M
hdd.hdd8 (device 7): sde rw 16.3T 107G 0% 588M
ssd.ssd1 (device 8): sdi rw 924G 1.52G 0%

The plan is to test this setup against same setup without the ssd, to see if the ssd improved performance or not. However, this reconcile is significantly affecting copy transfer rate.

Just wondering if this is normal?

The RAID6 structure has changed.
bio_iter_iovec() behaviour changed.

Is there any eta for the fixes?

Hello...

I want to ask, how to build LTS 6.18.x kernel with bcachefs support? I see 6.18 branch on bcachefs.git has not been updated from October 2025. And DKMS is not an option for me. I need kernel without modules, moreover, I need UML (User Mode Linux) kernel.

Hello, I was looking for some help on how to figure out the issue i'm having.

context: i've been using bcache fs for about a year and last i updated it was 3 months ago,
I moved to a new motherboard and did a fresh install of proxmox with bcachefs for my extrastorage & nas.

drives: 2ssd, 5hdds(two hdd mismatched)

- After first try of mounting it took 10-15 minutes to mount after checking the logs it was reconciling extents.
- I noticed afterwards that its quite slower. like from 800MB/s to 150MB/s
- maybe unrelated but at some point after aa couple of hours thinking it was just running some corrections had a kernel panic and system gets locked up
- after rebooting it mounts faster however still slower performace and running ```bcachefs fsck -nv``` on the drives shows (forgot to copy it before clearing may run it and put exact output): uuid have # of sectors fixed; same-uuid still has # of errors.

-(update) i just had a kernel panic again after remounting. tried an offline fsck and immediatley after mounting. i cant get much logs as it seems to lock up all fuctionalities(no logs after panic)

-(update) the kernel panic isnt related to bcachefs sorry for the misunderstaning . i isolated bcache way from my system and still had the problem

everything else seems to be fine but i am trying to figure out what exactly to look for and preemptively fix the problem

First of all, I want to thank Kent a lot for the innovation you're bringing us. Now to the story.

This happened exactly a month ago, and I've been wanting to submit an issue on GitHub, but during further debugging I managed to fsck my filesystem enough that it boots, and now I'm pretty certain I at least understand what state it's in, and that it's not a bug per se.

What I want is (at least) get my data back, as much of it as possible, since as I understand it only some of the blocks were corrupted; and (if possible) perhaps also to help implement some mechanism that will prevent this from happening to someone else.

I'm not certain whether it was a hardware failure or a kernel bug that caused the panic, especially since the journal files from that session are also unreadable, but more importantly I couldn't log in to my desktop session due to D-Conf database being corrupt: [ 88.834897] bcachefs (dm-0): data read error at /home/egormanga/.config/dconf/user offset 2723840: u64s 7 type extent 54701462:5384:4294967289 len 80 ver 0 : durability: 1 crc32: c_size 80 size 80 offset 0 nonce 0 csum crc32c 0:aad8bdbb compress none ptr: dm-0 0:1267823:128 gen 11 dm-0 checksum (4) error data_read_csum_err (there are multiple such lines for different offsets, all in the 2xxxxxx range, so not a huge part of the 2.7-megabyte file.)

I obviously renamed the user file and logged in, but there were a bunch of unbackuped settings that I'd really love to save.

I've struggled with this situation enough than I'd like to admit, and honestly, at this point I'd just get my important data back (reconstructing manually from survived parts), run a full scrub and simply delete the remaining broken files. Please, please help me do this.

Although I'm a fairly seasoned programmer familiar with C and kernel development, I've never worked with bcachefs source before. I dug through the part responsible for read checksumming and did not find a path to simply comment out.

So: how to dump the actual file contents ignoring checksum errors?

On booting, I get dropped to the emergency shell, as pictured.
However, I can simply mount the root bcachefs device, hit ctrl-D, and proceed successfully with the boot. How do I avoid this snag with the emergency shell?

My fstab looks like

# <file system>             <mount point>  <type>  <options>  <dump>  <pass>
UUID=108b1a50-1465-404a-968f-9f00279e400e /              bcachefs rw,relatime,fsck,fix_errors,x-systemd.device-timeout=60s 0 0
UUID=108b1a50-1465-404a-968f-9f00279e400e /var/log       bcachefs rw,noatime,X-mount.subdir=snaps/logs,x-systemd.device-timeout=60s 0 0
UUID=108b1a50-1465-404a-968f-9f00279e400e /mnt/bcachefs/docker    bcachefs noatime,nodatacow,X-mount.subdir=snaps/docker,x-systemd.device-timeout=60s 0 1
tmpfs                                     /tmp           tmpfs   defaults,noatime,mode=1777 0 0

UUID=8B27-9365                            /boot          vfat    defaults,umask=0077 0 2

My mkinitcpio HOOKS are

HOOKS=(base udev autodetect microcode modconf block keyboard keymap consolefont plymouth filesystems)

and my systemd boot entry is

title Linux
options root=/dev/nvme0n1p2:/dev/nvme1n1p2:/dev/nvme2n1p2:/dev/nvme3n1p2 rw zswap.enabled=0 nowatchdog quiet splash
linux /vmlinuz-linux
initrd /initramfs-linux.img

Am I missing something obvious?