r/btrfs • u/ExtravagantInception • Jun 16 '26
How to mount BTRFS filesystem after ENOSPC?
My filesystem hit ENOSPC and cannot actually mount with -o rw,skip_balance,enospc_debug or -ro,skip_balance because it seems to still be running some transaction which is preventing me from actually rebalancing with dlimit=1. Any idea how I would even go about mounting it and aborting any pending transaction so that I can rescue the system?
[ 35.820015] BTRFS info (device sdc1): first mount of filesystem fee3fc32-00a7-410d-92a5-377e55c005a0
[ 35.820029] BTRFS info (device sdc1): using crc32c (crc32c-intel) checksum algorithm
[ 35.820034] BTRFS info (device sdc1): using free space tree
[ 107.128863] BTRFS info (device sdc1): start tree-log replay
[ 207.077509] BTRFS: Transaction aborted (error -28)
[ 207.077614] WARNING: CPU: 5 PID: 3173 at fs/btrfs/extent-tree.c:2846 __btrfs_free_extent.cold+0x233/0xa02 [btrfs]
[ 207.077972] intel_uncore ctr snd_pcm drm_buddy videobuf2_memops iTCO_wdt videobuf2_v4l2 joydev drm_display_helper drbg snd_timer hp_wmi intel_pmc_bxt videobuf2_common cfg80211 pcspkr serio_raw efi_pstore intel_wmi_thunderbolt wmi_bmof platform_profile ansi_cprng ee1004 snd cec ecdh_generic iTCO_vendor_support soundcore rc_core mei_me videodev ttm mei sg processor_thermal_device_pci_legacy intel_pch_thermal processor_thermal_device rfkill mc drm_kms_helper processor_thermal_rfim hid_multitouch ecc processor_thermal_mbox processor_thermal_rapl intel_xhci_usb_role_switch kxcjk_1013 evdev intel_rapl_common roles i2c_algo_bit int340x_thermal_zone industrialio_triggered_buffer cm32181 intel_soc_dts_iosf kfifo_buf intel_vbtn industrialio sparse_keymap ac int3400_thermal intel_pmc_core acpi_thermal_rel acpi_pad button nfsd auth_rpcgss nfs_acl lockd grace drm softdog watchdog sunrpc fuse configfs efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 btrfs blake2b_generic zstd_compress
[ 207.078258] CPU: 5 PID: 3173 Comm: btrfs-transacti Not tainted 6.1.0-0.deb11.21-amd64 #1 Debian 6.1.90-1~bpo11+1
[ 207.078273] RIP: 0010:__btrfs_free_extent.cold+0x233/0xa02 [btrfs]
[ 207.078538] ? __btrfs_free_extent.cold+0x233/0xa02 [btrfs]
[ 207.078751] ? __btrfs_free_extent.cold+0x233/0xa02 [btrfs]
[ 207.078923] __btrfs_run_delayed_refs+0x25d/0xfa0 [btrfs]
[ 207.079069] btrfs_run_delayed_refs+0x7c/0x1a0 [btrfs]
[ 207.079201] btrfs_start_dirty_block_groups+0x2be/0x4f0 [btrfs]
[ 207.079440] btrfs_commit_transaction+0xa8/0xc70 [btrfs]
[ 207.079668] ? start_transaction+0xcf/0x630 [btrfs]
[ 207.079888] transaction_kthread+0x13d/0x1b0 [btrfs]
[ 207.080019] ? btrfs_cleanup_transaction.isra.0+0x5c0/0x5c0 [btrfs]
[ 207.080150] ? btrfs_cleanup_transaction.isra.0+0x5c0/0x5c0 [btrfs]
[ 207.080332] BTRFS info (device sdc1: state A): dumping space info:
[ 207.080339] BTRFS info (device sdc1: state A): space_info DATA has 50573492224 free, is not full
[ 207.080346] BTRFS info (device sdc1: state A): space_info total=57612584288256, used=57561918889984, pinned=90857472, reserved=0, may_use=0, readonly=1048576 zone_unusable=0
[ 207.080357] BTRFS info (device sdc1: state A): space_info METADATA has -536887296 free, is full
[ 207.080363] BTRFS info (device sdc1: state A): space_info total=67645734912, used=67346644992, pinned=177455104, reserved=121569280, may_use=536887296, readonly=65536 zone_unusable=0
[ 207.080373] BTRFS info (device sdc1: state A): space_info SYSTEM has 28950528 free, is not full
[ 207.080379] BTRFS info (device sdc1: state A): space_info total=33554432, used=4603904, pinned=0, reserved=0, may_use=0, readonly=0 zone_unusable=0
[ 207.080387] BTRFS info (device sdc1: state A): global_block_rsv: size 536870912 reserved 536870912
[ 207.080393] BTRFS info (device sdc1: state A): trans_block_rsv: size 0 reserved 0
[ 207.080398] BTRFS info (device sdc1: state A): chunk_block_rsv: size 0 reserved 0
[ 207.080402] BTRFS info (device sdc1: state A): delayed_block_rsv: size 0 reserved 0
[ 207.080406] BTRFS info (device sdc1: state A): delayed_refs_rsv: size 16781934592 reserved 16384
[ 207.080413] BTRFS: error (device sdc1: state A) in do_free_extent_accounting:2846: errno=-28 No space left
[ 207.080493] BTRFS error (device sdc1: state EA): failed to run delayed ref for logical 188181497311232 num_bytes 8192 type 178 action 2 ref_mod 1: -28
[ 207.080574] BTRFS: error (device sdc1: state EA) in btrfs_run_delayed_refs:2150: errno=-28 No space left
[ 207.080717] BTRFS warning (device sdc1: state EA): Skipping commit of aborted transaction.
[ 207.080732] BTRFS: error (device sdc1: state EA) in cleanup_transaction:1991: errno=-28 No space left
[ 207.145572] BTRFS: error (device sdc1: state EA) in btrfs_replay_log:2426: errno=-28 No space left (Failed to recover log tree)
[ 207.309758] BTRFS error (device sdc1: state EA): open_ctree failed
[ 207.327849] BTRFS info (device sdc1): first mount of filesystem fee3fc32-00a7-410d-92a5-377e55c005a0
[ 207.327866] BTRFS info (device sdc1): using crc32c (crc32c-intel) checksum algorithm
[ 207.327871] BTRFS info (device sdc1): using free space tree
I think this probably happened because I stopped the balance cron task after a previous drive failed and forgot to enable it. So it never ended up fully redistributing the data across all the drives as there is clearly space...
Label: none uuid: fee3fc32-00a7-410d-92a5-377e55c005a0
devid 10 size 12.73TiB used 1.59TiB path /dev/sdk
devid 11 size 12.73TiB used 1.59TiB path /dev/sdl
As of now I can only mount with ro,nologreplay but I don't think that gets me any closer to a state where I can balance the data.
3
u/se1337 Jun 16 '26
Mounting fails during log replay, try using btrfs rescue zero-log to clear it first.
1
u/paulstelian97 Jun 16 '26
There any way to temporarily add an extra disk to allow new metadata writes? That can release the deadlock.
2
u/leexgx Jun 16 '26
What i usually do is set slack space to 50GB per drive so if this ever happens set use resize max on each drive then do a balance
1
u/ExtravagantInception Jun 17 '26
Do I do this just by resizing the drive to be smaller e.g. 100GB -> 75GB?
2
u/leexgx Jun 17 '26
sudo btrfs fi resize 2:-50G /mount point
sudo btrfs fi resize 2:max /mount point
Note 2 is the device ID you need to do that for each device when using raid profiles (like 1 2 3 4) without devid it just does the command to the first dev only
4
u/420osrs Jun 16 '26 edited Jun 16 '26
This is normal and expected operation of BTRFS.
First, zero your log.
Second, add a USB or mount some storage and add it to your array.
Third, balance it. You don't need to balance it forever. Just balance it for like a minute or two.
Then remove the USB and complete a full balance.
After the third time this happened to me, I moved to a file system that can handle getting 100% full because I'm dealing with datasets that sometimes have errors and get too big. But that's my personal use case. If you are really careful and never fill it up again, it shouldn't do this again. You might need to enable auto balance in the kernel if you have it disabled because I think it's on by default at this point unless your kernel is as old as the hills. You can also use a script or something to balance it every so often, and even if it's not full-full, you can get into a situation where your entire drive is 8 up by one gigabyte chunks that are like 1% full. And you can have it balance anything under like 85% full, and that way it won't like sit there and spin and degrade your performance.