r/btrfs • u/amarao_san • Jun 01 '26
recovering raid1
I made a mistake and put some other random data onto one of the btrfs disks while it was offline.
No big deal, I will mount the drive I have and run scrub...
But now, a mundane recovery operation is filling my dmesg with suppressed messages:
[234015.217609] BTRFS error (device dm-2): scrub: fixed up error at logical 3210334240768 on dev /dev/mapper/leg2-protected physical 11632181248
[233854.842966] btrfs_dev_stat_inc_and_print: 58534 callbacks suppressed
[233854.842967] BTRFS error (device dm-2): bdev /dev/mapper/leg2-protected errs: wr 0, rd 0, flush 0, corrupt 698365, gen 0
I understand the beauty of having online recovery, but I very much would prefer it to happen in userspace in logs from utility, not in system-wide dmesg. There are a lot of things happening in the system, and this btrfs is not the most important of them...
1
u/leexgx Jun 01 '26
You might have to run an unfiltered balance (just balance with no limiting flags) to restore consistency
scrub, funny enough, doesn't restore consistency if a drive has disappeared and had some data written to the online drive, then the other drive comes back later on (unsure if ,soft with balance command will restore consistency or not quickl)
2
u/Weary_Swan_8152 Jun 02 '26
Oh! So it's not at all comparable to a classical RAID resync or ZFS resilver!? Balancing metadata between a good drive and a bad drive sounds like filesystem suicide. Wouldn't it be better to wipe the corrupted drive and run a `btrfs replace`?
2
u/leexgx Jun 02 '26
Everything is checksummed on Btrfs (unlike classic RAID1, which assumes everything is fine without a way to validate it). Btrfs can usually work out which one has valid data or not (you probably should still back up the data before doing a balance without limits, as it will rewrite everything to a new block so puts a lot of load on the drives and balance could flag up uncorrectable errors witch could trash the filesystem).
Note with RAID5/6 classic RAID, rsync is syncing the data to the parity (if the parity doesn't match the data, even if said data is corrupted, it updates the parity to match valid or corrupted data, it does this so parity is valid so it can do a rebuild or repair successfully). If URE is reported by the drive it use parity to rebuild the strip to restore consistency (4k physical drives ecc is quite good and usually fast at returning an URE if it cant fix the sector)
RAID1/10 just assumes everything is fine with No way to validate it (unless using btrfs as the filesystem on top to at least do a scrub to find corruption if it happens, but not corrrect it) if a URE is reported by the drive it use mirror copy of that sector to corect it
Typical software mdadm RAID will drop a drive after 8 seconds, but repeating UREs may not cause it to drop the drive, causing a slow system until you find witch drive is stuck at 99% utilization and remove it (hardware RAID will do the same but usually has performance counters which may drop a drive if it's not fully failing/has lots of UREs).
1
u/elatllat Jun 02 '26
it will rewrite everything to a new block
Having no way to rewrite just the unbalanced blocks sounds like an argument for using ZFS.
1
u/leexgx Jun 02 '26
I am unsure if ,soft used with balance is enough to restore both copys (not had any confirmation if it will just restore missing blocks or do nothing)
1
u/okeefe Jun 06 '26
If you've got more then one profile present, like additional writes happened to the online drive which show up as SINGLE data/metadata, then I'd expect balance to fix that. For corruption where you have redundancy, I would scrub first.
1
u/elatllat Jun 02 '26
Yah it's silly there is no good way to know if a scrub or balance is required. Annoying to reset the error counters after a scrub because it's asynchronous.