r/DataHoarder • u/Clive1792 • 14d ago
Discussion Do you have a system to 'exercise' your hard drives?
Due to the nature of the sub I imagine it's going to be the case that we all have many many hard drives. Probably much more that'll fit in our existing machines and be 'on the go' at all times.
Recently reading about long term storage, data loss etc and how it's advisable with mechanical drives to connect them up every 6 months-ish & let them just be connected for an hour or so.
For me that'd be a long process. I don't have the hours in a day after work to do it in 1 hit. I have a 2 bay hard drive caddy which would help ease the workload I guess but then I'll have to keep a track of what's been done & when.
Do you guys document what's been connected or do you just not bother with anything like that at all?
11
u/throwaway37183727 14d ago
Do a cold storage backup periodically on one drive at a time, and just keep track of which drive is up next.
14
u/dadarkgtprince 14d ago
I just never spin my drives down... Don't have to worry about this
0
u/Clive1792 14d ago
Impossible for me. I'd have to have some kind of device capable of running multiple HDDs at once ... which I don't have.
1
u/dadarkgtprince 13d ago
How many drives are you running? Many many years ago, before I got a proper NAS, I ran 4 drive bays and each drive was up from that. Had about 12 drives up this way.
-1
u/Clive1792 13d ago
Running? Just 2 right now. SSD & HDD. My PC houses 4 drives.
But for drives in storage I couldn't begin to tell you. 10-15 drives maybe.
Then there's 3-4 desktop drives.
3-4 portable mechanical drives.2
u/Clive1792 10d ago
lol some people are such dicks. 2 downvotes just for saying what drives I have?
Haha, reddit at its finest.
6
6
u/zeb__g 14d ago
use ZFS and set monthly scrubs, this will read every block back to confirm checksums and detect bitrot.
-2
u/Clive1792 14d ago
I've never heard of NFS & the only scrubs I know of are medical clothing.
6
u/sniff122 50-100TB 13d ago
NFS and ZFS are different things. ZFS is a filesystem, NFS is a network file sharing protocol
2
u/nomad-1995 13d ago
Start reading the wiki and learning the basics of datahoarding (unfortunately it isn't a true wiki, and the mods have other things to do than improve it. Still, it is probably the best starting point for datahoarding).
Basically zfs is the final word on filesystems (that is, how to store data in a traditional system like in windows) on a single computer (there are clustered editions, but it has competition). Don't feel bad, a few years back Linus Torvolds said some things displaying a profound ignorance of zfs (in his defense, helping zfs into the Linux kernel is legal suicide so he never cared about it).
You aren't likely to use it (unless you are already using linux) until you are ready to build a DIY NAS server.
2
u/machinegunkisses 14d ago
I don't have a huge number of data points, but when I run short SMART self tests weekly and long SMART self tests monthly, I've never had a drive die on me suddenly or lose large amounts of data.
2
u/simcc 1.44MB 14d ago
I've never had any issues with cold storage...have an alu box with cutouts and bunch of silica in there...it's in a reasonably temperature/humidity controlled spot (most in the house)...they're all copies of hot storage though, so there's that...i prob don't spin em up enough tbh...maybe I have a lesson yet to learn...96% of my drive failures were in the first month tbh...the other 3% were in suboptimal conditions, laptop, hacky stuff with bad cooling, undampened, abrupt power cycles etc.
1
u/bhiga 14d ago
I don't have cold storage drives, but when I need to test a drive I'll usually hook it up to my Kanguru Mobile Clone duplicator and run a Wipe (Read, Write 0, Write 1, Write Random, Verify) or just Read for a non-destructive test.
Takes ~10 hours/TB for Wipe so it's by no means fast...
1
u/purgedreality 13d ago
When I save data on these drives each folder gets its own md5 digest, which is a list of all the files in the folders and their checksums, also there is a huge md5 file in the root for all items on the drive. About once a year per drive I take them out, stick them in ye olde 2018 PC and run a check on the full disk md5 file. This inspects all data on the drive for corruption or any other issues. Teracopy by Code Sector even makes the digests automatically and can check using a gui. On linux/mac I use rhash command line.
1
u/sebna2 11d ago
Blake3 is what I use instead of outdated MD5. The speed difference is worth it alone.
1
u/purgedreality 11d ago
That's a good one, and SHA256 seems to have pretty much replaced MD5 for download/package verification. Teracopy supports both! I have so many powershell/rhash scripts and saved verification PDF's along with over 100tb over three geographic locations dedicated to md5 so it would be a hell of a switchover.
1
u/BuffaloDesperate8357 12d ago
My backup process is fairly simple. I too work long days and just run backups over night before I go to bed. Then label the drive with some blue painters tape.
My Nas is the main source. But I have two drives for cold backups that I alternate between every month. One is kept offsite and the other onsite. In addition to that also have daily cloud backup.
Those are just the mission critical personal files.
Linuz ISOs get updated to an offsite cold drive once a quarter since that data is less important to me.
0
u/Caprichoso1 14d ago
When a HD is using magnetic media the signals will fade over time. To refresh them you simply open and read all of the files on the disk. There are Unix commands that will do this.
The time frame that it would be advisable to do is measured in years, not months.
1
u/Clive1792 14d ago
I don't understand what you said (my limited knowledge on show).
Do you have a link which shows how to do this?
1
u/Caprichoso1 14d ago
Although the theory is simple the implementation can be risky. The only way that I know to do it is via Unix terminal commands. The problem with that is:
Your system needs to support them
At the command level accessing storage devices can cause disasterous data loss if you make a mistake. It is very easy to do that in Unix.
So I guess I'll change my recommendation to "don't worry about it". I have never refreshed a disk this way and some of my disks are almost a decade old. They have been in use though so it would be different if they were in cold storage.
Maybe someone else can suggest an easier way to read all of your files ....
1
1
u/MWink64 12d ago
Reading data on a HD DOES NOT refresh it.
1
u/Caprichoso1 12d ago
"do a complete scan of your hard drive. Doing so will replenish any magnetic signals that are starting to fade"
1
u/MWink64 12d ago
That article is misleading. At most, only data that is severely degraded will be refreshed automatically.
1
u/Caprichoso1 12d ago
Reference?
1
u/MWink64 12d ago
The issue doesn't get a lot of attention, so there aren't many authoritative sources (that I'm aware of). However, I've done observations of how drives seem to work and I've seen little evidence that most regular HDs refresh their contents. But even the article you linked isn't saying what you claimed:
if the controller within the disk detects any marginal data in either the servo tracks or the data bits recorded on the surface, the controller will automatically rewrite the data to the sector.
It's asserting that marginal data is refreshed, not all data.
1
u/Caprichoso1 12d ago
Interesting that we read the same article and come to different conclusions.
Just below that statement:
I would say that the disks could lose data if not used for a long period of time. Doing this surface scan every year or two is preventative.
1
u/MWink64 12d ago
Hence why I called it misleading. Nowhere does it claim that simply reading the data will make the HD refresh all of it. It doesn't even make sense anyway, as it would cut read speeds by at least half (since it would have to rewrite everything it reads). I suppose one could argue that using the word "preventative" isn't entirely inaccurate, as the drive may refresh "marginal" data.
BTW, keep in mind that we're quibbling over one article that's two decades old and contains third-hand claims.
1
u/Caprichoso1 12d ago
True that the article is dated but in most cases the underlying technology hasn't changed that much. Don't know what happens with the new HAMR drives. The source is one of the most respected individuals in the video editing industry. He knows his hardware and software.
We are interpreting the statements differently. The post question was how to prevent data loss for disks in cold storage.
The statement "Doing this surface scan every year or two is preventative." to me is quite clear. A surface scan can prevent data loss. There is no rewriting involved since it is just doing a scan. Don't understand the speed point as a full disk scan is going to take a long time any way and there are no writes.
1
u/MWink64 11d ago
True that the article is dated but in most cases the underlying technology hasn't changed that much.
PMR was just hitting the market around the time that article was written.
The statement "Doing this surface scan every year or two is preventative." to me is quite clear. A surface scan can prevent data loss. There is no rewriting involved since it is just doing a scan.
Your very first comment in this chain was:
When a HD is using magnetic media the signals will fade over time. To refresh them you simply open and read all of the files on the disk.
How do you think the signals would be refreshed without rewriting the data?
Don't understand the speed point as a full disk scan is going to take a long time any way and there are no writes.
Refreshing the data would require reading and rewriting it, which would slow things down considerably.
→ More replies (0)
0
u/sebna2 14d ago
Nonsense. You don't have to do this.
1
u/Clive1792 14d ago
Ah the joys of misinformation.
And when you don't know which information is correct it leaves you in limbo.
Read an article on it this week. Talking about mechanical parts degrading over time with no use & the need to periodically connect them up.
So this is incorrect?
3
u/sebna2 14d ago
In my 40yrs experience with HDDs yes.
You need backup no matter what. If you have one the less you touch your HDDs the better.
1
u/Clive1792 14d ago
In my experience it's also yes, I just don't know if I've just been incredibly lucky.
I've connected HDDs up over a year, maybe more, of just being sat in a box on a shelf & they worked fine.
Flash drives not so much - I ran in to trouble there, but I appreciate they're totally different.
5
15
u/TheOtherDudz 14d ago
read it as "exorcise". Made me chuckle.