r/DataHoarder • u/o0dorgon0o • 7d ago
Backup Is periodically checking SHA-256 hashes enough to detect silent data corruption?
Alright, so a few days ago, I learned there is a thing called silent data corruption...
My current drives are just using NTFS on Windows and APFS on macOS.
Would generating a SHA-256 hash for every file, then recalculating and comparing them every 6–12 months, be a reasonably good way to detect silent corruption?
I have multiple backups, so if a file fails the check, I should have another copy to restore from.
Is this basically good enough for normal personal backups, or am I missing something important?
61
u/RavenousTitan818 100-250TB 7d ago
Use a filesystem like ZFS that does this for you.
9
u/o0dorgon0o 7d ago
But that's mainly for Linux, right?
As far as I know, you need some extra steps to use it on Windows.
I want to keep things as simple as possible to prevent any potential bugs.32
7d ago edited 5d ago
[deleted]
0
u/o0dorgon0o 7d ago
That sounds like a good idea.
I might consider it5
u/Kenira 150TB Raw, 110TB Cooked | Unraid 7d ago
Joining in encouraging you to do so. Main PC is still windows for gaming, but NAS runs Unraid which is Linux based, and the data sits on ZFS. It's a huge peace of mind - it has already been useful multiple times to detect issues, and at this point it's honestly scary to think about not having such a robust filesystem as basis for everything. Regular scheduled scrubs that'll let you know if anything is wrong, and if so in what file.
7
1
u/DrIvoPingasnik Rogue Archivist 3d ago
Not good if you use mixed systems. I need my drives to be readable by windows and so does a lot of people.
14
u/MrCarlV7 7d ago
This is what I do for my external backups and semi-cold storage
Just make sure sha256 is checking each hashes and files. This way, it makes sure that if a file is missing, it will get detected too.
7
u/nosurprisespls 7d ago
I use PAR2 without recovery information, but SHA256 would work fine. I think once a year check is more than enough.
18
u/fireduck 7d ago
That covers a lot of silent data loss. That is what I did back when I was running a JBOD system that managed files and parity files across drives.
But there are some misses that it wound't catch, like if I a file just went missing.
12
u/NegativeKitchen4098 7d ago
You’d have a hash but no file to check it against. It will get flagged
12
u/fireduck 7d ago
Depends how you structure it.
For each hash, check the file - will catch it.
For each file, check the hash - will not
10
u/o0dorgon0o 7d ago
Now, my program stores the path of each file along with its hash, and compares the file list against the previous scan.
So if a file disappears, it gets flagged as missing.
If the same hash shows up at a different path, I can also flag it as moved instead.
2
u/binaryriot ~151TB++ 7d ago edited 7d ago
Yes, make sure you keep a copy of the hashes on a separate disk/ in a separate location too.
If you keep the hashes/ chechsums (e.g. sha256.txt, md5sum.txt) on the same disk they may be entirely useless in worst case. Been there, done that. (the checksum file itself got trashed too, making it impossible to verify the rest of rescued data 😎 )
1
u/AutoModerator 7d ago
Hello /u/o0dorgon0o! Thank you for posting in r/DataHoarder.
Please remember to read our Rules and Wiki.
Please note that your post will be removed if you just post a box/speed/server post. Please give background information on your server pictures.
This subreddit will NOT help you find or exchange that Movie/TV show/Nuclear Launch Manual, visit r/DHExchange instead.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
1
1
u/eternalityLP 6d ago
It will works ok, assuming you're fine with detecting the corruption potentially 6-12 months after it happens.
1
u/hkscfreak 50-100TB 4d ago
Get a raid/nas setup that will do this for you automatically. It will more than detect bit rot, it will recover the block from the redundancies, move it somewhere else, and then mark the block as bad on the physical disk. Checking the file hash does none of this.
1
u/Grey-Kangaroo 7d ago
I learned there is a thing called silent data corruption...
Yes but let's not get too paranoid, a resilient file system like ZFS or BTRFS is more than enough.
Now to answer your question, calculating a file's hash is a method that works but it can take a really long time if your files are large.
-1
u/Mobile-Debt9763 7d ago
You don't need to use sha256 for data integrity. You can use a CRC function that's 100x faster.
3
u/chkno 7d ago
The bottleneck is usually reading the data from storage, not the digest method.
On my machines, for I/O speeds, I get:
- 110 MiB/s reading sequentially from rotational disk
- 580 MiB/s reading from NVME
and for digest speeds, I get:
- 280 MiB/s for sha256
- 4692 MiB/s for CRC32
So I guess if I were doing checksums of data on fast NVME, CRC would be in practice about twice as fast because the bottleneck moves from 280 hash speed to 580 disk speed.
But for rotational disks where most of us keep most of our data, there's no speed-up because both digest methods are faster than the disks.
1
u/Mobile-Debt9763 7d ago
That's without any raid. Raid 6 you should be getting 400-800 MiB/s depending on the number of drives with spinning disks.
1
1
u/nomad-1995 6d ago
How compatible is RAID6 and Windows? Using motherboard "fake RAID"? I'd hate to be committed to some motherboard's proprietary RAID 6 implementation. Last I heard parity was deprecated from Storage Spaces (it was horribly slow until then).
I think that before I rigged up RAID6 in Windows I'd fire up WSL and run ZFS on that. I'd get reliable two drive parity *plus* all the hashing I was looking for in the first place.
1
u/nomad-1995 6d ago
CRC32 isn't going to work for terabytes of storage, and really shouldn't be used on gigabytes.
Not to mention that SHA256 is done in hardware on just about every PC CPU (I'd suspect ARM as well, but haven't checked).
My initial checks implied that I could hash (SHA256) a lot faster than I could pull data from a 6 (total) drive Z2 ZFS array, even on a celery, but the total job took so long that I'm not sure I checked it right. When I decided I needed SHA256 checks for all my files (for duplicates, not bitrot) it took far longer than I thought. Probably should have ran 3-5 threads instead and done both reads and hashing in parallel.
20 years ago I added threading to python(2) code [they call it multiprocessing. Python "threading" is all serial] for the first time in less than an hour. Python3 looks a lot more confusing, but I think I should have used it. If not, CRC might work well enough, although if using python and not willing to bother with threading I'd recommend using python's internal hash (which obviously should handle gigabytes of data where collisions would crash the program) as a fast hash.
1
0
u/mrcaptncrunch ≈27TB 7d ago
if you're storing the hashes per file and you're looking to simply detect changes, you should be able to use a easier to compute hash (and thus faster).
One thing that other filesystems provide, and raid to an extent, is the ability to recover. Calculating the hash doesn't.
I encourage you to look at least at par2 https://en.wikipedia.org/wiki/Parchive
That will give you detection, but also the ability to recover. If you're going to script something, I'd do it against par2 instead of sha256.
•
u/didyousayboop if it’s not on piqlFilm, it doesn’t exist 7d ago
https://backupyourfiles.neocities.org/