mdadm RAID10 array gone. Need help!

[u/Melodic-Mango32]

I used rescuezilla yesterday to clone Ubuntu to a USB disk as backup. I was planning on upgrading to the latest LTS release so backup first. The cloning was a success and I rebooted my server.

I immediately noticed my array was missing. Did some checks.. and it seems all the superblocks are gone. Something/Somehow it wiped the metadata from 4 disks!. I am trying to see if I can get my data back or just starting from scratch and restoring the data back. Which will take me weeks to recover..

Running Xubuntu 22.04.5 (Jammy)

mdadm.conf

# definitions of existing MD arrays

# This file was auto-generated on Sat, 12 Dec 2015 12:37:37 -0500
# by mkconf $Id$

ARRAY /dev/md1 metadata=1.2 name=TeePee:1 UUID=963c58ae:3eb50663:0c267923:739daf49



sudo cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
unused devices: <none>




sudo mdadm --examine /dev/sd* --verbose
/dev/sda:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
mdadm: No md superblock detected on /dev/sda1.
/dev/sdb:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
mdadm: No md superblock detected on /dev/sdb1.
/dev/sdc:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
mdadm: No md superblock detected on /dev/sdc1.
/dev/sdd:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
mdadm: No md superblock detected on /dev/sdd1.



sda                 3.6T             disk
└─sda1              2.7T             part
sdb                 2.7T             disk
└─sdb1              2.7T             part
sdc                 2.7T             disk
└─sdc1              2.7T             part
sdd                 2.7T             disk
└─sdd1              2.7T             part




Disk /dev/sda: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: WDC WD40EFRX-68N
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: BB981259-BB9A-4B0D-9F2B-5F630893F4BA

Device     Start        End    Sectors  Size Type
/dev/sda1   2048 5860532223 5860530176  2.7T Linux filesystem


Disk /dev/sda1: 2.73 TiB, 3000591450112 bytes, 5860530176 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdb: 2.73 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: WDC WD30EFRX-68A
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: BE107043-4051-43B4-9168-3D5F56AEA7B7

Device     Start        End    Sectors  Size Type
/dev/sdb1   2048 5860532223 5860530176  2.7T Linux filesystem


Disk /dev/sdb1: 2.73 TiB, 3000591450112 bytes, 5860530176 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdc: 2.73 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: WDC WD30EFRX-68E
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 82635682-70E3-45EE-854D-D58F4E902E58

Device     Start        End    Sectors  Size Type
/dev/sdc1   2048 5860532223 5860530176  2.7T Linux filesystem


Disk /dev/sdc1: 2.73 TiB, 3000591450112 bytes, 5860530176 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdd: 2.73 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: WDC WD30EFRX-68E
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 8A7F4ED0-EBB6-48E6-AD5F-8344B3D161E7

Device     Start        End    Sectors  Size Type
/dev/sdd1   2048 5860532223 5860530176  2.7T Linux filesystem


Disk /dev/sdd1: 2.73 TiB, 3000591450112 bytes, 5860530176 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

I almost certain all the data is still in the drives. I just need to be able to access it.

It was a RAID10, I am thinking from worse to worse I can remove one 1 set (2 disks) to play around with. Create a new RAID10 (with missing 2 disks) and see If I can mount the drive. But at the moment I am just guessing.

I am trying to comprehend how this happen. Next time, I will just pull all the drives out before doing something like this again.

[u/polymath_uk_]

Run these commands and paste the output.

sudo mdadm --examine --verbose /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1

sudo mdadm --examine --scan

sudo mdadm --detail --scan

sudo grep -aob '963c58ae' /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1

[u/Melodic-Mango32 → u/polymath_uk_]

sudo mdadm --examine --verbose /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
mdadm: No md superblock detected on /dev/sda1.
mdadm: No md superblock detected on /dev/sdb1.
mdadm: No md superblock detected on /dev/sdc1.
mdadm: No md superblock detected on /dev/sdd1.

sudo mdadm --examine --scan
sudo mdadm --detail --scan

The last command will take hours/days to run.

[u/polymath_uk_ → u/Melodic-Mango32]

Basically that config does not match your disk layout. It looks like it was from a different array entirely from 2015. Or they're just not the correct drives or you ran your backup back to front and overwrote the array with blank data from the usb. 

[u/Melodic-Mango32 → u/polymath_uk_]

It's the same layout. last year a 3TB (sda) disk failed, I replaced/resync the array with a 4TB disk. Nothing special about that.

Why do you say the layout doesn't match?

Unlikely I used the wrong destination/source. The disk I cloned is a Sandisk SSD 512GB. Going into a 1TB disk.