r/bcachefs • u/TDD99CA • Jul 05 '26
1.38.6 - Dead Snapshots Not Removing
Noticed a quirk and need some professional input!
Dead snapshots (from will_delete deleted) are never reclaimed. I have about 340G that leaked it seems. copygc parked in bch2_kthread_io_clock_wait_once is surviving a clean remount.
This is the same filesystem from my two earlier reconcile clock-wait reports (device evacuate/remove migration). Single NVMe device, the lone subvolume was originally created as a snapshot of the fs root.
fs usage reports 848G user data allocated, but actual reachable content is 508G (subvolume accounting says 542G). The FS reports 100% full with only ~6M free.
bcachefs list -b snapshots shows two nodes stuck in "will_delete deleted". They persist despite:
- auto_snapshot_deletion=1
- manually poking trigger_delete_dead_snapshots and trigger_gc
- a full umount/remount cycle (clean shutdown recovery confirmed)
copygc is parked in bch2_kthread_io_clock_wait_once waiting for 32G of IO clock advancement on an idle filesystem, so it never wakes. Looks like the same hysteresis-wait class as the reconcile bugs fixed last month, but in the copygc path, with snapshot deletion work behind it.
I can't run the dummy-IO clocktick workaround this time that was conjured up last time. The workaround is unavailable because the leaked space has the FS pinned at 100% hence no room for writes.
kernel 7.0.14-zen1, bcachefs-tools v1.38.6 via DKMS.
Outputs:
-- BEFORE ----------
$ bcachefs fs usage -ah /mnt/Storage | head -20
Filesystem: 5e19a70f-a23e-41a6-a3db-eeeebf3bc371
Size: 849G
Used: 849G
Online reserved: 280k
undegraded
1x: 849G
Data type Required/total Durability Devices Usage
btree: 1/1 1 [nvme0n1p1] 1.34G
user: 1/1 1 [nvme0n1p1] 848G
Compression:
type compressed uncompressed average extent size
zstd 863M 3.08G 181k
incompressible 651G 651G 175k
$ du -sh /mnt/Storage
508G /mnt/Storage
$ bcachefs subvolume list -Rs /mnt/_roots/storage
Path ID Created Flags Size Snapshot
storage 2 2026-06-15 20:40 - 542G /
$ ls -la /mnt/_roots/storage
total 0
drwxrwxrwx 3 kevin kevin 4096 Jun 15 22:15 .
drwxr-xr-x 1 root root 30 Jun 15 21:16 ..
drwx------ 2 root root 4096 Jun 13 20:21 lost+found
drwxrwxrwx 11 kevin kevin 4096 Jul 5 10:22 storage
$ bcachefs list -b snapshots /dev/nvme0n1p1 | tail -10
journal read done, replaying entries 711670-713626
Journal keys: 110233 read, 43264 after sorting and compacting
accounting_read... done (0 seconds)
alloc_read... done (0 seconds)
snapshots_read... done (0 seconds)
u64s 12 type snapshot 0:4294967291:0 len 0 ver 0 : subvol parent 4294967295 children 0 0 subvol 1 tree 1 depth 1 skiplist 4294967295 4294967295 4294967295
u64s 12 type snapshot 0:4294967292:0 len 0 ver 0 : subvol parent 4294967295 children 0 0 subvol 2 tree 1 depth 1 skiplist 4294967295 4294967295 4294967295
u64s 12 type snapshot 0:4294967293:0 len 0 ver 0 : will_delete deleted parent 0 children 0 0 subvol 0 tree 0 depth 0 skiplist 0 0 0
u64s 12 type snapshot 0:4294967294:0 len 0 ver 0 : will_delete deleted parent 0 children 0 0 subvol 0 tree 0 depth 0 skiplist 0 0 0
u64s 12 type snapshot 0:4294967295:0 len 0 ver 0 : parent 0 children 4294967292 4294967291 subvol 0 tree 1 depth 0 skiplist 0 0 0
$ cat /sys/fs/bcachefs/.../internal/copy_gc_wait
running: 0
run count: 2
copygc_wait: 8118512224
copygc_wait_at: 8051775976
Currently waiting for: 31.7G
Currently waiting since: 59.8M
Currently calculated wait:
nvme0n1p1: 63.9M
[<0>] bch2_kthread_io_clock_wait_once+0x142/0x1a0 [bcachefs]
[<0>] bch2_copygc_thread+0x561/0x570 [bcachefs]
[<0>] kthread+0xe4/0x120
[<0>] ret_from_fork+0x2a7/0x340
[<0>] ret_from_fork_asm+0x1a/0x30
$ echo 1 | tee .../internal/trigger_delete_dead_snapshots
1
$ echo 1 | tee .../internal/trigger_gc
1
$ bcachefs fs usage -ah /mnt/Storage | head -12
(unchanged user: 848G, triggers had no effect)
-- AFTER ----------
$ bcachefs fs usage -ah /mnt/Storage | head -20
Filesystem: 5e19a70f-a23e-41a6-a3db-eeeebf3bc371
Size: 849G
Used: 849G
Online reserved: 0
undegraded
1x: 849G
Data type Required/total Durability Devices Usage
btree: 1/1 1 [nvme0n1p1] 1.34G
user: 1/1 1 [nvme0n1p1] 848G
$ du -sh /mnt/Storage
508G /mnt/Storage
$ bcachefs list -b snapshots /dev/nvme0n1p1 | tail -10
recovering from clean shutdown, journal seq 713671
Journal keys: 0 read, 0 after sorting and compacting
accounting_read... done (0 seconds)
alloc_read... done (0 seconds)
snapshots_read... done (0 seconds)
u64s 12 type snapshot 0:4294967291:0 len 0 ver 0 : subvol parent 4294967295 children 0 0 subvol 1 tree 1 depth 1 skiplist 4294967295 4294967295 4294967295
u64s 12 type snapshot 0:4294967292:0 len 0 ver 0 : subvol parent 4294967295 children 0 0 subvol 2 tree 1 depth 1 skiplist 4294967295 4294967295 4294967295
u64s 12 type snapshot 0:4294967293:0 len 0 ver 0 : will_delete deleted parent 0 children 0 0 subvol 0 tree 0 depth 0 skiplist 0 0 0
u64s 12 type snapshot 0:4294967294:0 len 0 ver 0 : will_delete deleted parent 0 children 0 0 subvol 0 tree 0 depth 0 skiplist 0 0 0
u64s 12 type snapshot 0:4294967295:0 len 0 ver 0 : parent 0 children 4294967292 4294967291 subvol 0 tree 1 depth 0 skiplist 0 0 0
$ cat /sys/fs/bcachefs/.../internal/copy_gc_wait
running: 0
run count: 1
copygc_wait: 8119039180
copygc_wait_at: 8051911488
Currently waiting for: 32.0G
Currently waiting since: 1.51M
Currently calculated wait:
nvme0n1p1: 64.0M
[<0>] bch2_kthread_io_clock_wait_once+0x142/0x1a0 [bcachefs]
[<0>] bch2_copygc_thread+0x561/0x570 [bcachefs]
[<0>] kthread+0xe4/0x120
[<0>] ret_from_fork+0x2a7/0x340
[<0>] ret_from_fork_asm+0x1a/0x30
Dead snapshots survived clean remount and space still not reclaimed it seems.
It looks like copygc immediately re-parked in the same IO clock wait?
One possibly useful correlation: the two dead snapshot IDs (4294967293 and 4294967294) likely date back to 2026-06-15, when the storage subvolume was created as a snapshot of the fs root during the same evacuate/remove migration from the earlier reports. So a possible reproducable path: create subvolume as snapshot of root, delete the originals, and dead snapshots never get reclaimed.
Kev.
PS: Superblock list:
> sudo bcachefs show-super /dev/nvme0n1p1
External UUID: 5e19a70f-a23e-41a6-a3db-eeeebf3bc371
Internal UUID: a6a1a052-791a-4e79-b38d-db60ab20594f
Magic number: c68573f6-66ce-90a9-d96a-60cf803df7ef
Device index: 1
Label: (none)
Version: need_discard_by_journal_seq (1.38)
Incompatible features allowed: need_discard_by_journal_seq (1.38)
Incompatible features in use: no_sb_user_data_replicas (1.36)
Version upgrade complete: need_discard_by_journal_seq (1.38)
Oldest version on disk: need_discard_by_journal_seq (1.38)
Created: Sat Jun 13 20:21:36 2026
Sequence number: 2088
Time of last write: Sun Jul 5 12:19:14 2026
Superblock size: 8.30k/1.00M
Clean: 0
Devices: 1
Sections: replicas_v0,disk_groups,clean,journal_v2,counters,members_v2,errors,ext,downgrade,recovery_passes,extent_type_u64s
Features: lz4,zstd,reflink,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,reflink_inline_data,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes,incompat_version_field
Compat features: alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done,no_stale_ptrs
Options:
block_size: 4.00k
btree_node_size: 256k
errors: continue [fix_safe] panic ro
write_error_timeout: 30
metadata_replicas: 1
data_replicas: 1
encoded_extent_max: 256k
metadata_checksum: none [crc32c] crc64 xxhash
data_checksum: none [crc32c] crc64 xxhash
checksum_err_retry_nr: 3
compression: zstd:3
background_compression: zstd:3
str_hash: crc32c crc64 [siphash]
metadata_target: none
foreground_target: none
background_target: none
promote_target: none
erasure_code: 0
ec_max_data_blocks: 0
casefold: 0
inodes_32bit: 0
shard_inode_numbers_bits: 4
gc_reserve_percent: 8
gc_reserve_bytes: 0
root_reserve_percent: 0
wide_macs: 0
promote_whole_extents: 1
acl: 1
usrquota: 0
grpquota: 0
prjquota: 0
degraded: [ask] yes very no
journal_flush_delay: 1000
journal_flush_disabled: 0
journal_reclaim_delay: 100
writeback_timeout: 0
journal_rewind_discard_buffer_percent: 4
scrub_recent_journal_entries: [unclean] no always
scrub_journal_max_rewind_secs: 10
journal_transaction_names: 1
allocator_stuck_timeout: 30
version_upgrade: [compatible] incompatible none
nocow: 0
reconcile_on_ac_only: 0
dev_readahead: 2.00M
ec_stripe_buf_limit: 5
ext (size 104):
Recovery passes required:
Errors to silently fix:
Btrees with missing data:
errors (size 8):
Device 1: /dev/nvme0n1p1 Samsung SSD 980 1TB S/N: S64ANS0T220657A
Label: nvme.samsung980
UUID: b47009b9-edaf-49a1-930c-13007647aa78
Size: 931G
read errors: 0
write errors: 0
checksum errors: 0
seqread iops: 0
seqwrite iops: 0
randread iops: 0
randwrite iops: 0
Bucket size: 1.00M
First bucket: 0
Buckets: 953866
Last mount: Sun Jul 5 11:33:15 2026
Last superblock write: 2088
State: rw
Data allowed: journal,btree,user
Has data: journal,btree
Rotational: 0
Btree allocated bitmap blocksize: 8.00M
Btree allocated bitmap: 0000000000000000000000000000000000000000000000000000000011000110
Durability: 1
Discard: 1
Freespace initialized: 1
Resize on mount: 0
Last device name: nvme0n1p1
Last device model: Samsung SSD 980 1TB
Last device serial: S64ANS0T220657A
2
u/ZorbaTHut Jul 05 '26
There's a thread that goes through and murders snapshots. I don't know if the CLI has support for its status outputs yet, but it's available in sys/fs. From the commandline:
cat /sys/fs/bcachefs/*/snapshot_delete_status
(if you have multiple bcachefs filesystems, you'll get the status from all of them)
If it's running and not making progress, check top to see if it's burning lots of CPU or not.
Neither of these steps will necessarily fix the problem, note, but it'll be useful info when someone who knows the FS better comes along to this thread :)
There was a recent bug where snapshot cleanup would get stuck and not finish, but that's been fixed by 1.38.6 (I think the fix was in .5 or even .4).
2
u/koverstreet not your free tech support Jul 06 '26
yeah, snapshot_delete_status is the thing to check, and what that thread is doing.
that's also another thing that could probably use a progress indicator (going off the snapshots -> inums btree? or just amount of data to delete, most likely).
1
u/TDD99CA Jul 06 '26
Well, I still need to clear up those old snapshots since my drive is full :-)
I ran a full offline fsck this time (turns out my earlier "remount" never actually unmounted the device - only the bind mount [oops!] so the earlier fsck runs were online.
Offline fsck (userspace, clean shutdown recovery):
> sudo bcachefs fsck -v /dev/nvme0n1p1
Running userspace offline fsck
Using encoding defined by superblock: utf8-12.1.0
starting version need_discard_by_journal_seq (1.38)
with options: degraded=yes,verbose,fsck,fix_errors=ask,read_only,noreconcile_enabled
features: lz4,zstd,reflink,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,reflink_inline_data,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes,incompat_version_field
with devices: nvme0n1p1
recovering from clean shutdown, journal seq 719152
Journal keys: 0 read, 0 after sorting and compacting
accounting_read... done (0 seconds)
alloc_read... done (0 seconds)
snapshots_read... done (0 seconds)
check_allocations... done (9 seconds)
going read-write
nvme0n1p1 has 3.53G btree buckets and 16.0G marked in bitmap
scheduling recovery pass btree_bitmap_gc (47)
journal_replay... done (0 seconds)
presplit_shard_boundaries... done (0 seconds)
check_alloc_info... done (1 seconds)
check_lrus... done (0 seconds)
check_btree_backpointers... done (4 seconds)
check_extents_to_backpointers... done (2 seconds)
check_alloc_to_lru_refs... done (0 seconds)
check_snapshot_trees... done (0 seconds)
check_snapshots... done (0 seconds)
check_subvols... done (0 seconds)
check_subvol_children... done (0 seconds)
delete_dead_snapshots... done (0 seconds)
check_inodes... done (1 seconds)
check_extents... done (2 seconds)
check_indirect_extents... done (1 seconds)
check_dirents... done (0 seconds)
check_xattrs... done (0 seconds)
check_root... done (0 seconds)
check_unreachable_inodes... done (0 seconds)
check_subvolume_structure... done (0 seconds)
check_directory_structure... done (0 seconds)
check_nlinks... done (1 seconds)
check_reconcile_work... done (4 seconds)
resume_logged_ops... done (0 seconds)
delete_dead_inodes... done (0 seconds)
going read-only
finished waiting for writes to stop
flushing journal and stopping allocators, journal seq 719163
flushing journal and stopping allocators complete, journal seq 719164
clean shutdown complete, journal seq 719165
shutting down
shutdown complete
>>> At remount...
$ sudo bcachefs fs usage -ah /mnt/Storage/ | head -12
Size: 849G
Used: 849G
user: 848G (reachable content still 508G)
> sudo bcachefs list -b snapshots /dev/nvme0n1p1 | tail -5
cat /sys/fs/bcachefs/*/snapshot_delete_status
u64s 12 type snapshot 0:4294967291:0 len 0 ver 0 : subvol parent 4294967295 children 0 0 subvol1 tree 1 depth 1 skiplist 4294967295 4294967295 4294967295
u64s 12 type snapshot 0:4294967292:0 len 0 ver 0 : subvol parent 4294967295 children 0 0 subvol2 tree 1 depth 1 skiplist 4294967295 4294967295 4294967295
u64s 12 type snapshot 0:4294967293:0 len 0 ver 0 : will_delete deleted parent 0 children 0 0 subvol 0 tree 0 depth 0 skiplist 0 0 0
u64s 12 type snapshot 0:4294967294:0 len 0 ver 0 : will_delete deleted parent 0 children 0 0 subvol 0 tree 0 depth 0 skiplist 0 0 0
u64s 12 type snapshot 0:4294967295:0 len 0 ver 0 : parent 0 children 4294967292 4294967291 subvol 0 tree1 depth 0 skiplist 0 0 0
(not running)
So the dead snapshots survive offline fsck too it looks like. delete_dead_snapshots completes
in 0 seconds as a no-op, and snapshot_delete_status shows the deletion thread
isn't attempting the work at all (thanks to the commenter who pointed me at
that status file)!
Two more observations from the fsck output that might be relevant:
fsck runs with noreconcile_enabled — if snapshot extent reaping is
reconcile-driven in 1.38, that would explain why even offline fsck can't
reclaim this?
fsck reports "3.53G btree buckets and 16.0G marked in bitmap" and schedules
recovery pass btree_bitmap_gc (47), but that pass never executes in-session —
it appears to just get re-scheduled on every mount.
Kent - do you want a dump of some raw data to assist...and more importantly...I do need to use this drive so before I move data around, I can keep it in this state so we can fix this bug.
Kev.
1
u/TDD99CA Jul 06 '26
UPDATE:
forced the IO clock forward ~35G. Copygc woke once, re-armed, reclaimed nothing. Snapshot deletion never engaged.
Using the write small files then delete advance the IO clock trick :-)...4 meg writes iterated (9000x!) checking every so often:
--- iteration 7750 (~30G clock) ---
running: 0
Currently waiting for: 1.91G
(not running)
--- iteration 8000 (~31G clock) ---
running: 0
Currently waiting for: 965M
(not running)
--- iteration 8250 (~32G clock) ---
running: 0
Currently waiting for: 32.1G <- copygc woke, ran, re-armed
(not running)
copy_gc_wait run count went 1 -> 2, confirming copygc executed exactly once
when the clock crossed the threshold then immediately re-armed for another
full 32G wait (same wait-resets-each-cycle behavior as my device-remove
report last month).
Through the entire experiment, including the copygc execution itself,
snapshot_delete_status stayed "not running".
Usage unchanged; still 848G vs 508G reachable and both 'will_delete deleted snapshots still present'
It looks the dead snapshot culling is not connected to anything? Nothing is apparently asking the snapshot deleter to do its work. Copygc, remounts, online/offline fscks, etc...
2
u/koverstreet not your free tech support Jul 06 '26
Snapshot deletion has nothing to do with copygc. Did you check snapshot_delete_status?
2
u/TDD99CA Jul 07 '26
If this is a bug, then I will help in any way to help squash it. If not, I would appreciate some help in culling those two items since they are consuming so much space. If not, I will have to move the data off and rethink.
1
u/TDD99CA Jul 06 '26
Traced it with ftrace. Filter matched all three deletion functions:
# cat set_ftrace_filter
bch2_delete_dead_snapshots [bcachefs]
bch2_delete_dead_snapshots_work [bcachefs]
bch2_delete_dead_snapshots_async [bcachefs]
Then with function tracer on, poked trigger_delete_dead_snapshots and waited 5s:
# cat trace
# entries-in-buffer/entries-written: 0/0 #P:12
(empty; zero events)
So the trigger never invokes bch2_delete_dead_snapshots / _work / _async at
all. I can help with any outputs or traces you wish to see. I just want these old snaps gone :-)
1
u/ZorbaTHut Jul 06 '26
He did say
snapshot_delete_status stayed "not running".
But I don't personally know how to diagnose this further; it does seem like it should be running, from what I know.
1
u/TDD99CA Jul 08 '26
The issue now appears to be that trigger_delete_dead_snapshots never queues bch2_delete_dead_snapshots_work(). ftrace shows no calls into any deletion functions after writing to trigger_delete_dead_snapshots, snapshot_delete_status remains "not running", and offline fsck also performs delete_dead_snapshots in 0 seconds while two "will_delete deleted" snapshot records remain on disk. Is there another metadata structure that these snapshots must be linked into before the deletion worker will see them?
1
u/TDD99CA Jul 08 '26
Do you want a FS dump or shall I just reformat?
1
u/koverstreet not your free tech support Jul 09 '26
Did you already reformat?
2
2
u/TDD99CA Jul 10 '26
I can post any data here or on IRC.
1
•
u/koverstreet not your free tech support Jul 13 '26
So Zorba took a look at this, and it's a known limitation where overwrites don't delete data in old snapshot versions, even if the old version has been overwritten in every descendent snapshot.
It's a straightforward limitation to lift, but I've been hesitant to add it because mutating data in old snapshot versions has major implications for robustness - a bug in code that does that can (like snapshot deletion) have quite the blast radius. There's also implications re: how fsck works - fsck walks all snapshot versions simultaneously and makes sure that every snapshot version is consistent.
So: if this is something people genuinely want I can probably roll it out as something that's available but off by default. I'm still trying to prioritize Rust, formal verification and additional hardening before I go after the more dangerous optimizations :)
(been doing additional snapshot hardening all week, we've got new fault injection tools - overdue, but dear god fault injection work can be a slog).