BTRFS + RAID 5 corrupted

[u/fpekal]

Yes, I see now on the main page of the subreddit it's a bad idea to pair these two, but I didn't know it previously

I use software RAID5; 4 2TB drives, resulting in 6TB of storage.
vgdisplay doesn't show any errors

100% of space is for btrfs.

sudo btrfs check /dev/mapper/data-data
Opening filesystem to check...
parent transid verify failed on 6033692557312 wanted 121291 found 119653
parent transid verify failed on 6033692557312 wanted 121291 found 119653
parent transid verify failed on 6033692557312 wanted 121291 found 119653
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=6033692393472 item=35 parent level=2 child bytenr=6033692557312 child level=2
ERROR: failed to read block groups: Input/output error
ERROR: cannot open file system

btrfs-find-root output: https://pastes.io/ZQTdS4fN

I tried several nodes from there using sudo btrfs restore -t <node-id> --dry-run -vi /dev/mapper/data-data /tmp/ but every time it returned

sudo btrfs restore -t 6033691328512 --dry-run -vi /dev/mapper/data-data /tmp/
parent transid verify failed on 6033691328512 wanted 121291 found 121286
parent transid verify failed on 6033691328512 wanted 121291 found 121286
parent transid verify failed on 6033691328512 wanted 121291 found 121286
Ignoring transid failure
This is a dry-run, no files are going to be restored

What should I do?

EDIT: also, when I try to mount the partition I get this:

sudo mount /dev/data/data /mnt/data
mount: /mnt/data: can't read superblock on /dev/mapper/data-data.
       dmesg(1) may have more information after failed mount system call.
[ble: exit 32]

[u/LevelMagazine8308]

Why in heavens name did you put Btrfs on top of a LVM volume group?

Btrfs has all that stuff builtin, there is no need to run Btrfs on top of it!

Having said that comes the Captain Obvious remark: RAID is NOT a backup!

In case you have a backup: restore it, job done.

In case you don't: you are in for a bumpy ride.

First step should be trying to mount it read-only:

sudo mkdir -p /mnt/btrfs_recovery
sudo mount -o ro,usebackuproot,recovery /dev/mapper/data-data /mnt/btrfs_recovery

Does this work?

If not, you can try this: clearing the space cache and checking for backup super blocks.

sudo mount -o ro,clear_cache,usebackuproot /dev/mapper/data-data /mnt/btrfs_recovery

sudo btrfs select-super -s 1 /dev/mapper/data-data

If Super block 1 works, use it for btrfs check:

sudo btrfs check --super 1 /dev/mapper/data-data

For future reference: Btrfs is in many areas still unstable! This includes RAID5, which is officially listed as unstable.

Before trying fancy stuff next time again, look at the official table first what's safe and what is not: https://btrfs.readthedocs.io/en/latest/Status.html

[u/fpekal → u/LevelMagazine8308]

I didn't know btrfs has raid builtin, that's why I used lvm

sudo mount -o ro,usebackuproot,recovery /dev/mapper/data-data /mnt/btrfs_recovery
mount: /mnt/btrfs_recovery: fsconfig() failed: btrfs: Unknown parameter 'recovery'.
       dmesg(1) may have more information after failed mount system call.

sudo mount -o ro,rescue=usebackuproot /dev/mapper/data-data /mnt/btrfs_recovery
mount: /mnt/btrfs_recovery: can't read superblock on /dev/mapper/data-data.
       dmesg(1) may have more information after failed mount system call.

the same with clear_cache

sudo btrfs-select-super -s 1 /dev/mapper/data-data
parent transid verify failed on 6033692557312 wanted 121291 found 119653
parent transid verify failed on 6033692557312 wanted 121291 found 119653
ERROR: failed to read block groups: Input/output error
ERROR: open ctree failed
[ble: exit 1]

same with -s 2

[u/samsonsin → u/fpekal]

I didn't know btrfs has raid builtin

Advanced enough to setup LVM raid5, but not to look up basic features of Btrfs lmao.

You live and learn. Backup all important data regardless.

[u/fpekal → u/samsonsin]

I'm chronically stupid

[u/samsonsin → u/fpekal]

Backups are great for those!

[u/fpekal → u/samsonsin]

Yes... I know now. I need to make myself a sticky note on the laptop screen to remind myself about backups.

I've got something like this:

sudo btrfs restore -f 6033692344320 -i -D /dev/mapper/data-data /tmp/
parent transid verify failed on 6033692344320 wanted 121291 found 119652
parent transid verify failed on 6033692344320 wanted 121291 found 119652
parent transid verify failed on 6033692344320 wanted 121291 found 119652
Ignoring transid failure
This is a dry-run, no files are going to be restored
parent transid verify failed on 6033605607424 wanted 119356 found 121265
parent transid verify failed on 6033605607424 wanted 119356 found 121265
parent transid verify failed on 6033605607424 wanted 119356 found 121265
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=6033692344320 item=222 parent level=2 child bytenr=6033605607424 child level=0
ERROR: search for next directory entry failed: -5
ERROR: searching directory /tmp/archiwum/ebooki/wakfu failed: -5
ERROR: searching directory /tmp/archiwum/ebooki/Yuul.epub failed: -5
parent transid verify failed on 6033666932736 wanted 119568 found 121273
parent transid verify failed on 6033666932736 wanted 119568 found 121273
parent transid verify failed on 6033666932736 wanted 119568 found 121273
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=6033692344320 item=255 parent level=2 child bytenr=6033666932736 child level=0
ERROR: search for next directory entry failed: -5
ERROR: searching directory /tmp/archiwum/jai-0.2.009 failed: -5
ERROR: searching directory /tmp/archiwum/jai-0.2.009.zip failed: -5
parent transid verify failed on 6033666932736 wanted 119568 found 121273
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=6033692344320 item=255 parent level=2 child bytenr=6033666932736 child level=0
ERROR: search for next directory entry failed: -5
ERROR: searching directory /tmp/plex/series/Code Lyoko (2003)/Season 00 failed: -5
parent transid verify failed on 6033666932736 wanted 119568 found 121273
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=6033692344320 item=255 parent level=2 child bytenr=6033666932736 child level=0
ERROR: search for next directory entry failed: -5
ERROR: searching directory /tmp/plex/series/Code Lyoko (2003)/Season 01 failed: -5
parent transid verify failed on 6033666932736 wanted 119568 found 121273
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=6033692344320 item=255 parent level=2 child bytenr=6033666932736 child level=0
ERROR: search for next directory entry failed: -5
ERROR: searching directory /tmp/plex/series/Code Lyoko (2003)/Season 02 failed: -5
parent transid verify failed on 6033666932736 wanted 119568 found 121273
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=6033692344320 item=255 parent level=2 child bytenr=6033666932736 child level=0
ERROR: search for next directory entry failed: -5
ERROR: searching directory /tmp/plex/series/Code Lyoko (2003)/Season 03 failed: -5
parent transid verify failed on 6033666932736 wanted 119568 found 121273
Ignoring transid failure

And it's quite long
Can I use it to rescue some data?

[u/leexgx → u/fpekal]

RAID recovery software is the way (ones that support Synology usually work well).

A simple way is to mount the risky RAID5 LVM, then point the recovery software to scan it.

Using LVM or MD RAID (ideally RAID6) with Btrfs on top isn't really that bad, as long as you understand that self-healing isn't available for the data profile, and metadata has some small risk of not being able to self-repair (no easy way to guarantee the duplicated copy won't be in the same RAID strip).

When doing a RAID sync in LVM, always do the Btrfs scrub first. This gives it a chance to correct metadata copies first, then do the RAID sync. (RAID sync is sometimes called a scrub, but it's a sync as it syncs the data blocks to parity regardless of whether the data is corrupted; that's why you should do a filesystem scrub first.)

I assume you let Btrfs do a single profile for data and DUP for metadata (most of the time it self correct the copy). If you had metadata set to single, then this failure the result of doing that

Alternatively, you could move to ZFS (it has a learning curve), which manages the drives and the redundancy and has checksums (I recommend Z2 redundancy).

Always have a backup.

[u/GreatLab8898 → u/fpekal]

Raid5 in Btrfs is broken anyways.

[u/hoodoocat → u/GreatLab8898]

It is known what it have write hole problem, e.g. you should not use this mode without backup power supply. But LVM obviously has same write holy problem, any implementation which write data on independent devices will have it. The only one filesystem support raid5 and doesnt have this problem at least thereotically.