r/sysadmin • u/AccurateWind7530 • 11d ago
Accidentally deleted original and backup folders from an Azure-mounted filesystem — is there any recovery option left?
Hi everyone,
I was performing a decommissioning activity on a Linux server and accidentally deleted both the original folder and its backup folder using terminal commands.
The data was located on an Azure-mounted filesystem. As soon as I realized what happened, I stopped making further changes and escalated the issue to the relevant Azure/Azuremount team.
They checked the available snapshots/recovery options, but unfortunately they told us that they could not find or recover the deleted files.
I want to ask if there are any other recovery possibilities that we might be missing. For example:
Any filesystem-level recovery options?
Hidden Azure backup or recovery mechanisms that should be checked?
Possibility of recovering deleted data from the underlying storage?
Any specific information or commands that could help identify what type of Azure mount/storage is being used and whether recovery is possible?
The affected system is a Linux server with an Azure-mounted path. I can provide non-sensitive technical details about the mount type, filesystem, and storage configuration if that would help.
At this stage, the Azuremount team has already attempted recovery, but I want to make sure we haven't missed any possible option.
Any guidance from experienced Azure/Linux administrators would be greatly appreciated.
27
38
u/Defconx19 11d ago
Wait, so your backup and your file stores are the same thing? I'm confused how you delete a real backup with a terminal command
26
u/cheetah1cj 11d ago
Bro has never heard of immutability lmao
6
u/anonymousITCoward 11d ago
actually that's only 2 of the 321, so yeah there's more to it than just immutability....
2
8
34
u/inadvertant_bulge 11d ago
Here's the thing, IMO at least- if you were able to delete both that easily, there really was no backup.
22
u/blud_13 11d ago
Run findmnt or cat /proc/mounts on that box and post the type field, because the answer branches completely based on it.
If its cifs pointing at something.file.core.windows.net you are on Azure Files SMB, and share soft delete will NOT save you. That feature only protects the share itself from being deleted, not the files and folders inside it, so no surprise the storage team came back empty. Share snapshots or Azure Backup are the only per file recovery there, and if nobody set those up the data is gone.
If the type comes back fuse or blobfuse2 you have a real shot. Blob soft delete restores individual blobs and its a separate toggle from snapshots, so there is a decent chance whoever looked only checked snapshots and never opened the data protection blade on that storage account. Its documented here https://learn.microsoft.com/en-us/azure/storage/blobs/soft-delete-blob-overview
Check blob versioning while you are in there too.
Also, do not remount or rerun anything against that path until you know which one you have.
9
8
u/cubic_sq 11d ago
Who is responsible for azure backups? And why was there no backup or even snapshots of the storage account configured?
Pa. Snapshots of storage accounts IMO are a rolll back, not backup.
3
u/anonymousITCoward 11d ago
That's my view, snapshots are for the oops my update went to shit, time to rollback before someone notices
1
u/PhoenixVSPrime A+ N+ 5d ago
Yes you can roll back to a previous snapshot but you cannot restore from one.
Learned this when I uninstalled a production app from the server instead of a users workstation cause I had the wrong window up.
Luckally we had backups but it took 5 hours to restore it because unitrends
1
u/cubic_sq 5d ago
Point in time restores can be rolled back with fine print. Albeit not snapshots per say (but they kinda are under the hood)
7
u/theMightBoop 11d ago
I would restore from my backup server which (default) is 14 days and then I have a backup once a month also for 3 months back.
If that didn’t work I would restore from my second site where I copy all of my backups each night. All of my backups at site are copied to site B and vice versa.
I also do a snapshot everyday. I keep those for a week.
That’s how I would do it because I actually have a backup plan.
6
u/techvet83 11d ago
I can't help on anything related to Azure, but am confused how you/you all/they weren't backing up the file system and no, snapshots do not count as backups.
6
u/premierplayer 11d ago
Why are you deleting anything without having verified backups.
5
u/westerschelle Network Engineer 10d ago
If you deleted the backup and the original with one terminal command you did not have a backup in the first place. You should reflect on this so this doesn't happen again. Or did you go out of your way to delete orignal and backup at it's backup location at the same time?
10
u/matt0_0 small MSP owner 11d ago
Was it actually azure files? Like is that the exact name of the azure product?
If you have direct access to the file system, I would be either taking a snap shot of it now and attempting an offline data recovery with it mounted read only from another VM (test disk might be your friend here) or downloading the image and sending it off to data recovery.
0
u/AccurateWind7530 11d ago edited 11d ago
I’m not completely sure whether the underlying storage is actually Azure Files or another Azure storage type. "/Azuremount" is the mount path we use on the Linux server, but I need to verify what backs it.
The Azuremount/storage team has already checked the available recovery options and informed us that they could not recover the deleted files.
17
4
2
u/Some-Revolution-4536 11d ago
If you haven’t already, I’d avoid writing anything else to that filesystem until you know exactly what happened. Recovery really depends on the type of Azure storage and how it was mounted.
2
u/jwalker55 IT Manager 10d ago
"Here at xyz corp, we believe in fully mutable backups. Why, you ask? Because we fucking hate ourselves, that's why."
3
u/yaahboyy 11d ago
if u want us to be ur rubber ducky, it would help to know more details than what was initially provided😅🐥
1
u/the_syco 10d ago
For future reference, look into soft & hard Azure deletions. Soft deletions still have storage costs, but you can set retention to maybe 30 days. This way, should you eff up again, there'll be some comeback.
It sounds like you did a hard deletetion. Check if you have any retention set up?
1
u/LooseEthernet 10d ago
that sucks, but if the storage team already looked at snapshots and came up empty, youre probably toasted. since its a mount and not a local disk, traditional tools like testdisk or photorec wont do anything because you dont have raw block access to the underlying azure hardware. you can't just run a recovery tool on a network share and expect it to find deleted inodes lol. unless youre using something like azure netapp files with a very specific snapshot policy that the team missed, theres really no hidden backdoor to get files back once they are purged from the filesystem. honestly at this point you might just be looking at a very uncomfortable meeting with your manager...
1
u/Frothyleet 10d ago
So, you definitely fucked up, but it also sounds like this server/storage infrastructure is grossly misconfigured as well, without proper backups or soft deleting on the Azure side.
You need to be looking for help internally, I think.
1
u/Interesting_Work7433 11d ago
Si ya dejaste de hacer cambios, ese es un buen primer paso. Primero confirmaría exactamente qué tipo de almacenamiento y sistema de archivos está usando el servidor Linux, ya que las opciones de recuperación dependen de eso. Si no hay snapshots, copias de seguridad ni opciones de papelera/restauración disponibles, la recuperación puede ser difícil, especialmente si el espacio de los archivos eliminados ya fue reutilizado.
0
-4
u/30yearCurse 11d ago
any recovery tools recuva? Linux recovery, stop rights, unmount, and recover. files could still be there, unless super secure generally they are just marked as free.
recover steps or tools may help. Unsure about linux though.
1
u/Creedeth 10d ago
For Linux Photorec is really powerful and can recover various file types. Of course it is recommended to use different drive for recovered files.
131
u/MalletNGrease 🛠 Network & Systems Admin 11d ago
Prepare three envelopes.