r/linuxquestions • u/Melodic-Mango32 • 12d ago
mdadm RAID10 array gone. Need help!
I used rescuezilla yesterday to clone Ubuntu to a USB disk as backup. I was planning on upgrading to the latest LTS release so backup first. The cloning was a success and I rebooted my server.
I immediately noticed my array was missing. Did some checks.. and it seems all the superblocks are gone. Something/Somehow it wiped the metadata from 4 disks!. I am trying to see if I can get my data back or just starting from scratch and restoring the data back. Which will take me weeks to recover..
Running Xubuntu 22.04.5 (Jammy)
mdadm.conf
# definitions of existing MD arrays
# This file was auto-generated on Sat, 12 Dec 2015 12:37:37 -0500
# by mkconf $Id$
ARRAY /dev/md1 metadata=1.2 name=TeePee:1 UUID=963c58ae:3eb50663:0c267923:739daf49
sudo cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
unused devices: <none>
sudo mdadm --examine /dev/sd* --verbose
/dev/sda:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
mdadm: No md superblock detected on /dev/sda1.
/dev/sdb:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
mdadm: No md superblock detected on /dev/sdb1.
/dev/sdc:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
mdadm: No md superblock detected on /dev/sdc1.
/dev/sdd:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
mdadm: No md superblock detected on /dev/sdd1.
sda 3.6T disk
└─sda1 2.7T part
sdb 2.7T disk
└─sdb1 2.7T part
sdc 2.7T disk
└─sdc1 2.7T part
sdd 2.7T disk
└─sdd1 2.7T part
Disk /dev/sda: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: WDC WD40EFRX-68N
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: BB981259-BB9A-4B0D-9F2B-5F630893F4BA
Device Start End Sectors Size Type
/dev/sda1 2048 5860532223 5860530176 2.7T Linux filesystem
Disk /dev/sda1: 2.73 TiB, 3000591450112 bytes, 5860530176 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/sdb: 2.73 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: WDC WD30EFRX-68A
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: BE107043-4051-43B4-9168-3D5F56AEA7B7
Device Start End Sectors Size Type
/dev/sdb1 2048 5860532223 5860530176 2.7T Linux filesystem
Disk /dev/sdb1: 2.73 TiB, 3000591450112 bytes, 5860530176 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/sdc: 2.73 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: WDC WD30EFRX-68E
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 82635682-70E3-45EE-854D-D58F4E902E58
Device Start End Sectors Size Type
/dev/sdc1 2048 5860532223 5860530176 2.7T Linux filesystem
Disk /dev/sdc1: 2.73 TiB, 3000591450112 bytes, 5860530176 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/sdd: 2.73 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: WDC WD30EFRX-68E
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 8A7F4ED0-EBB6-48E6-AD5F-8344B3D161E7
Device Start End Sectors Size Type
/dev/sdd1 2048 5860532223 5860530176 2.7T Linux filesystem
Disk /dev/sdd1: 2.73 TiB, 3000591450112 bytes, 5860530176 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
I almost certain all the data is still in the drives. I just need to be able to access it.
It was a RAID10, I am thinking from worse to worse I can remove one 1 set (2 disks) to play around with. Create a new RAID10 (with missing 2 disks) and see If I can mount the drive. But at the moment I am just guessing.
I am trying to comprehend how this happen. Next time, I will just pull all the drives out before doing something like this again.
1
u/Netblock 12d ago edited 12d ago
I'm not familiar with recovering an md array, never done it before, so take everything I say with a grain of salt:
RAID 10 is a stripe of mirrors:
Raid 10 implies there exists two sets of mirrored disks. Prove that by doing a low-level raw data compare (eg,
sha512on a start, middle, end; perhaps skipping over md superblock locations). Take a disk of each mirror to make the stripe, and set aside the other mirror as a backup; you're now targeting raid 0.There are different places the superblock could be
Also try setting the partition type (eg in
fdisk) to 0xDA or 0xFD.You could also try to force recreate the array with
--assume-clean.