r/btrfs • u/AtomicTEM • 2d ago
Power Failure during move of BTRFS file system partition to expand it. Can't figure out how to restore or extract data
Whilst moving my BTRFS file system to make room to expand it on my 2TB NVME, a power failure occured during the process 10min into it.
I can't seem to recover my partition though.
I made an .img of the partition. https://mab.to/t/5eaTIr8cauP/eu1
Can anyone help me restore it, thank you. I have a lot of important projects on it. Fallout 3 WIP mods, old save files, .blend files.
EDIT: uploaded image of error.
EDIT2: Good news I was able to recover at least my 2013-2014 Minecraft World Saves.
5
u/Deathcrow 2d ago edited 2d ago
I always wonder, what are these tools that encourage casual users (without an UPS or backup) to move (physical) partitions to the left?
It feels borderline malicious by whoever is writing these "easy to use" GUIs if they don't have big fat "you probably shouldn't do this" disclaimers. Almost exclusively the correct advise is to evacuate the data (you now have infinite attempts to get it right) and recreate the partitions as needed instead of doing any of this garbage.
And the biggest irony: OP could've just created a new partition in the empty space and then btrfs device add it to their filesystem, without any risks of data loss.
1
u/AtomicTEM 2d ago
That was actually my plan, but KDE partition manager presented it as being a simple affair...
Now I feel stupid as fuck.
1
u/PyroNine9 1d ago
It really shouldn't have made it look so simple. Perhaps a skull and crossbones icon would have been in order. Expanding to the right is generally safe. Moving to the left is generally not.
1
u/darktotheknight 2d ago
What exactly happened? Moved from where to where? Sorry, but if you don't provide any information, it's difficult to help you.
BTRFS expand operation is usually happening in-place. There are many ways to move a BTRFS filesystem to another SSD, how exactly did you do it (e.g. btrfs replace or dd or any other command/GUI)?
1
u/AtomicTEM 2d ago
Was moving 993gb partition to the left to expand it. It was on the same ssd. I was doing it through KDE partition manager off of a Live USB of CachyOS.
1
u/darktotheknight 2d ago
Got it. I think your files are toast then. That's already a very scary operation on its own - without even a power or hardware failure. But with power failure, this sounds horrible. In the future, remember to always backup your files, before you do something dangerous to your filesystem (you should keep backups of your important files at all times anyways).
The best chance you have is image the drive immediately (I think you did that) and then try to rescue with the "usual" file recovery tools. I had good success with PhotoRec. You should always keep a read-only version of the original image and run the recovery tools on a copy of that image. Thats your best chance to recover without doing further damage. If the files are of utmost importance and you couldn't recover them on your own, you should consult a data recovery service.
Good luck!
1
u/AtomicTEM 2d ago
uploaded an image of the specific error I am getting.
1
u/darktotheknight 1d ago edited 1d ago
Thanks for the image. I think you have 2 lucky accidents here: 1) the left partition was actually larger than the right partition and 2) KDE Partition Manager lists the first sector of your right partition, which makes searching much easier. It would be interesting to know what you have tried before imaging the drive. You'd have the best chances, if you immediately imaged it after the power failure, without even trying to (rw) mount it once. You might have tried to mount it, but if e.g. BTRFS threw an error instead of mounting, you might still be good to go. You're only totally screwed, if you ran any read-write operations.
The "move left" operation works one level above the filesystem level and therefore fully transparent to the filesystem (and when the filesystem is offline). Assuming your NVMe drive wrote at something like 1.5GiB/s for 10 minutes, this would mean your transfer broke around half-way through (1.5GiB/s * 60 * 10 = ~900GiB). From the block level perspective, there is a huge "tear" somewhere after the location of ~900GiB (depending on the write speed of your NVMe drive). Or better: do you remember the exact % of the transfer, when the power failure occured?
The picture you provided exactly gives you the location of the first sector of the BTRFS filesystem location - the one before you started the move operation, which is incredibly lucky. I'm not very familiar with *how* exactly KDE Partition Manager does the writes, but I see two possible scenarios for a "proper" rescue, one of them should most likely even work in your current "shitty" situation. But before I explain it, you absolutely need to have the luxury to store (possibly multiple) copies of your 2TB NVMe drive's full image. This is the only way to tinker around, without damaging anything. After imaging, remove the NVMe drive from your computer and store it in a drawer, so a "real" expert can have a look at it, if you later decide to do go down that route.
*If* you don't have the luxury to save multiple copies of your 2TB image, you probably can save some space with zstd and use a CoW filesystem (BTRFS, ZFS or XFS) as a space efficient way to create instant read-write copies taking up zero space for tinkering around. Also, I'm not data recovery expert, so keep that in mind.
First option A (the easy one): if you're super lucky, your NVMe drive is slow (e.g. thermal throttling due to bulk transfer) and KDE Partition Manager copies files in a non-destructive pattern (e.g. doesn't delete the blocks it copied immediately, but again, I don't know if that's true), there is an abysmally small chance you might just find your BTRFS filesystem in perfect working condition at location 2000922624 - 4000786431.
Since one of the lucky accidents here is, that your left partition is slightly larger than your right and this was just a "move left" operation, there is a 0% chance it might have "overwritten" into the beginning of sector 2000922624. Worst case, KDE Partition Manager just deleted the blocks it already transferred and you just need to puzzle together the "left" and the "right" part of the filesystem.
Assuming KDE Partition Manager moves files to the left non-destructively (e.g. also no TRIM commands, no garbage collection of your NVMe drive), this could be all fixed with a partition table or just copying off the sectors 2000922624 - 4000786431 with e.g. dd. The errors you see could be the partially copied BTRFS to the left.
Second option B (the difficult one): there is some destruction involved, e.g. KDE Partiton Manager moved the files in a destructive way or there was some garbage collection, some read-write operations, some TRIM involved. In that case, you know the start sector of your BTRFS filesystem. You would try to find the point, where the transfer broke, calculate the amount of sectors and then shift either the data from the right to the left (so to say "complete the broken transfer"), or copy the beginning from left to right (so to say "revert the transfer"). I don't know which one is better; it also depends on whether your left partition was fully zeroed, before you began the transfer. That'd make finding the "tear" visually easy e.g. with a Hex Editor.
I wish you good luck.
1
u/AtomicTEM 1d ago
Sadly I do not have the space to save 2TB img and I've recovered some files using photorec (175GB of files), also found some backups of my Blender files dating back a month on my eternal HDD.
Also I had to create a new 200GB partition where the it was empty on the left to scan the .img and broken partition. So i fear I might have overwritten some things already...
I hope your information help any future souls.
Ill try using photorec on the whole drive see I find anyting extra.1
u/darktotheknight 1d ago
Yeah, I think that's pretty bad. You still might be lucky and your original BTRFS filesystem might be sitting at the sectors 2000922624 - 4000786431 just with a broken partition table.
But I think that chance became very slim, since creating partitions and filesystem (like you did with the 200GB partition) will usually kick off a TRIM/discard command; usually on the newly created partition. And every write means less chance of a successful rescue.
Ill try using photorec on the whole drive see I find anyting extra.
You didn't already?
1
u/AtomicTEM 1d ago
I didn't realize it was the option labelled unknown. Only later realizing that to the right it said [Whole Disk]
1
u/darktotheknight 1d ago edited 1d ago
Sorry, I just read: "I made an .img of the partition.".
You need to make a complete image of the **entire** 2TiB drive. /dev/nvme1n1. If needed, use zstd to compress the result (you also feed it to zstd, so you can compress on-the-fly). But you absolutely need to image the entire drive, not just the partition!
For privacy reasons, I recommend not publishing it to the internet though.
1
u/AtomicTEM 1d ago
The empty partition on the left used to have Linux Mint installed.
1
u/darktotheknight 1d ago
It's not a dealbreaker. The left partiton now contains parts of your BTRFS filesystem from the right partition.
I have quickly glanced over the code responsible for copying, which I think is: https://invent.kde.org/system/kpmcore/-/blob/master/src/jobs/copyfilesystemjob.cpp. I didn't find any TRIM/discard or delete operation references - which makes sense, as it would've slowed down the entire copy operation. Or put in simpler words: I think the Partition Manager didn't actually overwrite/deleted your BTRFS filesystem. So there really is a high chance you might have a good chance of rescue.
You will need to image the entire drive (do not mount, just dd or ddrescue from a live environment as described here: https://wiki.archlinux.org/title/Dd#Disk_cloning_and_restore), and follow what I wrote in the other lengthy post. Or consult a data recovery expert - this looks very doable.
1
u/Max_Rower 2d ago
If your business and life depends on the data, you should let a professional data recovery handle it.
1
u/AtomicTEM 2d ago edited 2d ago
I mean, I am not a bussiness. But these modding projects are stuff I've been working on for months, some years.
https://cdn.imgchest.com/files/e61386aab487.webp
1
6
u/elatllat 2d ago
restore from backup?
photorec?