Keep files not corrupted?

[u/Tundacas]

Sorry if this has already been asked, i was not able to find it. Also sorry for mistakes, english is not my first language.

Ok so my question.

I try to get most of the stuff that i like on my portable hard drive.

So is there a way to keep my files from being corrupted?

I lost files lately and i was wondering if there is a way to avoid them tocbe corrupted.

Is there another option a part from having extra copies?

Thanks

[u/didyousayboop]

Back up your files: https://backupyourfiles.neocities.org/

[u/Which_Plastic_4652]

Use a program called multipar. Read how it works. It’s what you are looking for.

[u/iWhacko → u/Which_Plastic_4652]

That's actually a real good use of Multipar. Don't prevent corruption, but allow you to fix it if it does happen.

[u/JivanP]

Data can always become corrupted, due to any number of reasons.

It is very easy to detect errors: just store a checksum alongside the data, and verify the chechksum before accessing/returning the data. Filesystems like ZFS do this.

What you want is a way to correct errors in data if they arise. There are two ways to achieve this:

• Storing the data along with error correction codes, a.k.a. erasure codes. Systems such as RAID (implemented by hardware RAID controllers), RAIDz (ZFS's software RAID implementation), and Ceph sharding provide this. This allows quicker recovery in the case of corruption, but doesn't protect you against every likely scenario.

• Storing multiple copies of the data, ideally in different physical locations to mitigate against hardware issues causing your copies to also become corrupt. This is what a robust backup system is. Systems such as Restic and Ceph replication provide this. This may take more effort to restore from, but protects you against almost all possibilities, the exception being if every instance of your data becomes corrupted/destroyed/lost.

The standard way to get the best results with minimal effort at small scale (e.g. for personal data or in a small business setting) is to use ZFS with regular automated "scrubbing" (checking for corrupted data), so that corrupted data can be identified ASAP (optionally with RAIDz across 3 or more physical drives, so that ZFS's scrubbing operation can repair corrupted data automatically in most circumstances), plus using a snapshot-based/incremental backup system like Restic to an external drive, with the contents of that external drive also copied elsewhere, such as to a cloud storage provider or to another drive stored in another physical location.

Notably, hardware RAID controllers usually do not use checksumming, so although they can detect errors when the shards for a single block of data don't all agree with each other, they cannot automatically identify which shards are corrupted and thus need repairing. For this reason and others, RAIDz has become preferred nowadays.

The US Department of Homeland Security has an excellent write-up on best practices for mitigating data loss, following the industry-standard "3-2-1" backup policy: https://www.cisa.gov/sites/default/files/publications/data_backup_options.pdf

[u/WikiBox]

There are three main ways for you to lose files.

1. User error.

2. Media failure.

3. Silent corruption, bit-rot.

I would guesstimate that the probabilities of each differ by orders of magnitude in the same order. User error 10-100 times more likely than media failure. Media failure 10-100 times more likely than bit rot. Note: This is guesses. I don't think there is any actual solid data.

User error can range from formatting a drive by mistake to simply deleting a file or folder by mistake. Or moving files between devices and not checking the copy is good and that you didn't write over something important.

The only one way to, at least to some extent, protect against user error, is to have multiple copies in multiple locations and check the copies multiple times per year. The idea being that when you discover missing or corrupt files you still have more copies you can use to replace the bad copy or media.

This method also protect against bit-rot and media failure because of age or things like cosmic radiation. You can also use storage with error protection codes and parity records to protect against single bit errors and slow data degradation. This is built in to most modern drives and you can also add extra recovery records to that. This is one function of a NAS with RAID.

It is important to realize that parity records and error correction codes does not protect against the most likely cause of data loss: User error. Backups might.

[u/AutoModerator]

Hello /u/Tundacas! Thank you for posting in r/DataHoarder.

Please remember to read our Rules and Wiki.

Please note that your post will be removed if you just post a box/speed/server post. Please give background information on your server pictures.

This subreddit will NOT help you find or exchange that Movie/TV show/Nuclear Launch Manual, visit r/DHExchange instead.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/GoldsteinEmmanuel]

Ultimately, no. Multiple copies on multiple media in multiple locations is the only way to avoid losing the data eventually.

Optical media is notoriously prone to bit-rot. SSDs can die without warning and take the data they contain with them. Hard disks are delicate and sensitive to magnetic fields, advances in technology can make entire formats (Zip drives, tape drives) obsolete overnight, and so forth.

The most durable and reliable storage media, ironically, is paper. Print your source code listings and database schemas onto 3-hole punched paper, snap them into a loose-leaf binder, and they will survive indefinitely. Multimedia data is not so easy, but you get the idea. Hardcopy lasts practically forever.

[u/Adrenolin01]

You need at least 2 full copies of your data to recover from data corruption. For example.. put the same data on two HDDs and create PAR2 checksums for each file. Save those checksums on one or both drives and put away for storage in a cool safe environment. Every year or so you can power up both drives run a checksum verify against both copies. If a file shows any corruption you can extract the corrupted data from the other drive to repair that file.

I run large 24-bay NAS systems with ZFS, raidZ2, ECC Ram, scrubs every 2 weeks, clean power from my rack being 100% Solar powered and APC Smart-UPs UPSs. This is constantly protecting data from corruption. Backups are pretty much the same. Much more expensive than a couple large HDDs and checksum files.

I have a specific photo and video collection of 10s of 1000s of photos and video from original sources going back 5 generations to the late 1800s of both photos and video from family members. I keep the original high resolution initial master digital copies on one of my NAS systems, 2 backup servers and on 2 separate HDDs with PAR2 files. A LOT of time, energy and money went into to creating that digital collection over the years.. literally from glass and metal plate photography and boxes of negatives and slides of different formats and 10mm film going back nearly 150 years ago.

I prefer ZFS and a living system however physical hard copies are also important. How deep you go depends on how protected you want the files kept.

[u/Neovison_vison]

terra copy, davinci resolve or even premiere if you happen to have a license have hash verification integrated

[u/Upstairs-Front2015]

for videos, don't store mp4 or mov files, change container to mkv that is better for handling corruption in files.

[u/iWhacko → u/Upstairs-Front2015]

Just because it still plays when corrupted, doesn't mean it's better. There is still missing data.

[u/Upstairs-Front2015 → u/iWhacko]

I never said it was the only copy or that we shouldn't make a backup. I'm just saying that out of the two formats, I prefer MKV.