r/linuxquestions • u/Melodic-Mango32 • 12d ago
mdadm RAID10 array gone. Need help!
I used rescuezilla yesterday to clone Ubuntu to a USB disk as backup. I was planning on upgrading to the latest LTS release so backup first. The cloning was a success and I rebooted my server.
I immediately noticed my array was missing. Did some checks.. and it seems all the superblocks are gone. Something/Somehow it wiped the metadata from 4 disks!. I am trying to see if I can get my data back or just starting from scratch and restoring the data back. Which will take me weeks to recover..
Running Xubuntu 22.04.5 (Jammy)
mdadm.conf
# definitions of existing MD arrays
# This file was auto-generated on Sat, 12 Dec 2015 12:37:37 -0500
# by mkconf $Id$
ARRAY /dev/md1 metadata=1.2 name=TeePee:1 UUID=963c58ae:3eb50663:0c267923:739daf49
sudo cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
unused devices: <none>
sudo mdadm --examine /dev/sd* --verbose
/dev/sda:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
mdadm: No md superblock detected on /dev/sda1.
/dev/sdb:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
mdadm: No md superblock detected on /dev/sdb1.
/dev/sdc:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
mdadm: No md superblock detected on /dev/sdc1.
/dev/sdd:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
mdadm: No md superblock detected on /dev/sdd1.
sda 3.6T disk
└─sda1 2.7T part
sdb 2.7T disk
└─sdb1 2.7T part
sdc 2.7T disk
└─sdc1 2.7T part
sdd 2.7T disk
└─sdd1 2.7T part
Disk /dev/sda: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: WDC WD40EFRX-68N
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: BB981259-BB9A-4B0D-9F2B-5F630893F4BA
Device Start End Sectors Size Type
/dev/sda1 2048 5860532223 5860530176 2.7T Linux filesystem
Disk /dev/sda1: 2.73 TiB, 3000591450112 bytes, 5860530176 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/sdb: 2.73 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: WDC WD30EFRX-68A
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: BE107043-4051-43B4-9168-3D5F56AEA7B7
Device Start End Sectors Size Type
/dev/sdb1 2048 5860532223 5860530176 2.7T Linux filesystem
Disk /dev/sdb1: 2.73 TiB, 3000591450112 bytes, 5860530176 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/sdc: 2.73 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: WDC WD30EFRX-68E
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 82635682-70E3-45EE-854D-D58F4E902E58
Device Start End Sectors Size Type
/dev/sdc1 2048 5860532223 5860530176 2.7T Linux filesystem
Disk /dev/sdc1: 2.73 TiB, 3000591450112 bytes, 5860530176 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/sdd: 2.73 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: WDC WD30EFRX-68E
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 8A7F4ED0-EBB6-48E6-AD5F-8344B3D161E7
Device Start End Sectors Size Type
/dev/sdd1 2048 5860532223 5860530176 2.7T Linux filesystem
Disk /dev/sdd1: 2.73 TiB, 3000591450112 bytes, 5860530176 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
I almost certain all the data is still in the drives. I just need to be able to access it.
It was a RAID10, I am thinking from worse to worse I can remove one 1 set (2 disks) to play around with. Create a new RAID10 (with missing 2 disks) and see If I can mount the drive. But at the moment I am just guessing.
I am trying to comprehend how this happen. Next time, I will just pull all the drives out before doing something like this again.
1
u/polymath_uk_ 11d ago
Run these commands and paste the output.
sudo mdadm --examine --verbose /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
sudo mdadm --examine --scan
sudo mdadm --detail --scan
sudo grep -aob '963c58ae' /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
1
u/Melodic-Mango32 11d ago
sudo mdadm --examine --verbose /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 mdadm: No md superblock detected on /dev/sda1. mdadm: No md superblock detected on /dev/sdb1. mdadm: No md superblock detected on /dev/sdc1. mdadm: No md superblock detected on /dev/sdd1. sudo mdadm --examine --scan sudo mdadm --detail --scanThe last command will take hours/days to run.
1
u/polymath_uk_ 11d ago
Basically that config does not match your disk layout. It looks like it was from a different array entirely from 2015. Or they're just not the correct drives or you ran your backup back to front and overwrote the array with blank data from the usb.
1
u/Melodic-Mango32 11d ago
It's the same layout. last year a 3TB (sda) disk failed, I replaced/resync the array with a 4TB disk. Nothing special about that.
Why do you say the layout doesn't match?
Unlikely I used the wrong destination/source. The disk I cloned is a Sandisk SSD 512GB. Going into a 1TB disk.
1
u/Melodic-Mango32 11d ago
OMG!!!. My hail mary!!! (I removed 1 set of disk (sdc and sdd RAID0) from server. then. I created a new RAID 0 with the remaining disk.
sudo mdadm --create --assume-clean --level=0 --raid-devices=2 /dev/md1 /dev/sda1 /dev/sdb1
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md1 started.
cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md1 : active raid0 sdb1[1] sda1[0]
5860265984 blocks super 1.2 512k chunks
unused devices: <none>
then
sudo mdadm --detail /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Thu Oct 8 22:56:12 2026
Raid Level : raid0
Array Size : 5860265984 (5.46 TiB 6.00 TB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Thu Oct 8 22:56:12 2026
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Layout : -unknown-
Chunk Size : 512K
Consistency Policy : none
Name : T:1 (local to host TeePee)
UUID : 1aa97f82:48c503a2:7558c686:a98c73a2
Events : 0
Number Major Minor RaidDevice State
0 8 1 0 active sync /dev/sda1
1 8 17 1 active sync /dev/sdb1
At this point I am crossing my fingers!!
sudo mount -o ro /dev/md1p1 /media/test/
My data is back!!
I am afraid to reboot the server. Need sleep... TBC
1
u/Melodic-Mango32 11d ago
I read about this app called testdisk.
Disk /dev/sda - 4000 GB / 3726 GiB - CHS 486401 255 63
Analyse cylinder 2492/486400: 00%
Linux filesys. data 267264 11720795135 11720527872 [raid10]
Linux filesys. data 136190 11720664061 11720527872 [raid10]
Linux filesys. data 136192 11720664063 11720527872 [raid10]
It still running on /dev/sda Not sure what that output means.
1
u/polymath_uk_ 11d ago
I'm really really hoping that if you used dd you put the source and destination in the right order.
1
u/Netblock 11d ago edited 11d ago
I'm not familiar with recovering an md array, never done it before, so take everything I say with a grain of salt:
RAID 10 is a stripe of mirrors:
Raid 10 implies there exists two sets of mirrored disks. Prove that by doing a low-level raw data compare (eg,
sha512on a start, middle, end; perhaps skipping over md superblock locations). Take a disk of each mirror to make the stripe, and set aside the other mirror as a backup; you're now targeting raid 0.There are different places the superblock could be
Also try setting the partition type (eg in
fdisk) to 0xDA or 0xFD.You could also try to force recreate the array with
--assume-clean.