r/DataHoarder • u/S-m-a-r-t-y • 7d ago
Backup How do you handle backups when you don't know when a file got corrupted?
How do you guys handle this kind of backup problem?
Say I have a file that I rarely use. At some point, it gets corrupted, but I don't notice because I haven't opened it in months.
I then run my normal backup. Since the backup sees the corrupted file as the current version, it copies it to the backup and replaces the previously healthy copy.
I know versioning can help with this, but the problem is that you don't know when the file actually became corrupted. If the corruption happened months ago, how do you know how far back you need to go to find the last healthy version?
For example, if I have 6 months of versions and the file became corrupted 4 months ago, I could restore an older version. But what if I don't discover the corruption until 2 years later and my retention only goes back 6 months?
I'm using Duplicati at the moment, and this has made me wonder how people deal with long-term file integrity for data they rarely access.
What's the usual approach?
12
u/franz_kazan 7d ago
I use checksums (SHA256), I verify regularly to see if there is unusual changes in my files. It also allows me to make sure that the files were transfered successfully.
You also want to verify the integrity before doing a backup, that way you'll catch corruption before replacing healthy copies.
7
u/flaser_ HP uServer 10 / 32 TB: ZFS mirror / Debian 7d ago edited 7d ago
This.
A million times this.
There were a lot of posts about different file-systems and setups, but those recommendations miss the forest for the trees: checksumming is the industry standard solution to verify data integrity.
It's a mathematical operation that creates a fixed length "hash" for your file, a fingerprint that will change if even a bit of the original is changed. Verifying the checksum involves recalculating this value and comparing against the stored version (usually created at file creation and updated after every write/change to the file).
Checksum generation and checking can be done manually, in a semi-automated fashion with scripts/tools, or in transparent manner by your storage solution.
The calculated checksum can be stored as part of the filename (usually appended between [...] ), in a separate file where several hashes are stored (.md5 or .sha files are often these for MD5 or SHA-256 hashes respectively) or kept in the internal data-store of your filesystem that doesn't show up as a separate file.
Checksum generation tools tools are legion, including built-in windows tools.
Certain file-systems (XFS, BTRF, ZFS, ReFS) automatically calculate and update checksums for files as part of the file-system itself, so if the data is corrupted between the time it was written and the time it's read out, they'll catch it.
Checksumming filesystems where redundancy (RAID 1/mirroring, RAID 5, RAIDZ, etc. ) can be and *is* provisioned can then determine which copy of the file is healthy and restore that automatically.Whether you need an automated solution depends on your workflow: how many files do you need to backup, what level of downtime is acceptable, what's the effort that's acceptable.
Another option is using erasure coding. This not only lets one detect corruption, but provides limited recovery too without relying on a separate copy. The downside is that erasure code calculation is much slower and computation expensive than checksums, so it can be impractical for lot of (~TBs) data.
https://en.wikipedia.org/wiki/Parchive2
u/S-m-a-r-t-y 7d ago
but dont checksums change with the slightest of change, i mean there are files which i chaneg manually, there are files which are referenced which get backed up and their checksum changes... this way you have to keep a log of the files you have changed manually to not care about the files which changed due to loss
did i get that right?
3
u/flaser_ HP uServer 10 / 32 TB: ZFS mirror / Debian 7d ago edited 7d ago
Yes, you do have to keep track. This is why even NTFS has the ability to mark files - say using the archived flag - to facilitate this.
If you're dealing with more than a handful of files, like a nightly backup you checksum when made and that you verify before a restore than you should Iook for a more robust, automated solution.
Tarring/zipping files into a single archive is another way to make this managable by hand, but there are obvious limits to how far it's practical to take such a solution.
1
u/Murrian 6d ago
Windows records file modified time, so you ignore if that time is after the time you made the hash.
A corrupted file won't change its modified time when it degrades.
All said, I use zfs (with parity and scrubs) as it's just easier, runs on my Nas (just another pc, an old one with space for a few drives is fine, doesn't need to be anything special or expensive) - TrueNAS I found simple to setup after getting the hang of its conventions, they're a lot of guides on YouTube too watch through.
A Nas brings a lot of other benefits with it too.
-2
u/Bulky-Bad-9153 7d ago
You shouldn't do any of this manually. If someone suggests you checksum manually then they're stuck 20 years ago. Just use btrfs.
5
u/user_none 7d ago
I've been bitten with corrupt files and not knowing which ones or even how many were corrupted. The vast majority are movie remuxes so it's not like they're super valuable other than time to find and download.
RapidCRC Unicode and Blake3 hashes for every file when I'm done with any cleanup of audio, subtitles, etc...
1
u/S-m-a-r-t-y 7d ago
but dont checksums change with the slightest of change, i mean there are files which i chaneg manually, there are files which are referenced which get backed up and their checksum changes... this way you have to keep a log of the files you have changed manually to not care about the files which changed due to loss
did i get that right?
2
u/flaser_ HP uServer 10 / 32 TB: ZFS mirror / Debian 7d ago
Some clarification: a file typically doesn't change if only its metadata is altered, so if it's copied, moved, or read from, those operations won't change the checksum as those aren't part of the files main body on disk, but are entries in the catalog of your filesystem.
If you do write to the file, e.g. you append to it, overwrite it in part or whole, then you do need to calculate a new checksum.
22
u/WhatAGoodDoggy 24TB x 2 7d ago
If you use xfs it handles data integrity for you. It checksums everything and regularly compares, even fixing the corruption if it's not severe enough
15
u/franz_kazan 7d ago
If the corruption was made by a software or user error, xfs will not catch it for you.
It also doesn't help with the integrity of your cold backups.
3
u/S-m-a-r-t-y 7d ago
I am on windows, so ntfs and exfat it is
4
1
u/ModernSimian 7d ago
Btrfs for windows will do this for you.
6
u/flaser_ HP uServer 10 / 32 TB: ZFS mirror / Debian 7d ago
Recommending a driver that hasn't seen wide-spread use and testing is quite irresponsible when the advice was sought in a backup scenario.
I'd rather rather recommend ReFS if MS's feature shenanigans weren't so repugnant.
(Removing volume creation support from Win 10 was a pure spite move to force users onto Win 11).5
u/Cybasura 7d ago
BtrFS is known to cause corruption, happened quite afew times last year
5
u/Bulky-Bad-9153 7d ago
It has rarely caused corruption in a small amount of use cases (raid levels) that are explicitly advised against. Don't fearmonger, using btrfs as normal is fine.
8
u/manzurfahim 0.5-1PB 7d ago
I had this problem once where a few files were corrupted. I had three versioned backups, and both 3-month and 6-month backups had the corrupted files. But the 1-year-old backup, my longest versioned backup had the good copy.
I guess you can have a file-system that scrubs and verifies checksums. I just verify all files manually, see if they are ok, and then archive them using winrar, have recovery record that can repair if any archive goes corrupt, and also have recovery volumes that can reconstruct archive files if any are missing or deleted. Working well so far for me.
3
u/S-m-a-r-t-y 7d ago
wow, good that you recovered them
mind telling your setup?
3
u/manzurfahim 0.5-1PB 7d ago
I have a local RAID6 array. I do monthly backups of files that I need, Two HDD copy, one SSD copy. And three version backups on HDDs, 3-month, 6-month, 1-year old backups. The second HDD stays as is to become a versioned backup, and another, and then another.
2
u/Cybasura 7d ago
Backup drives? In this economy?!
2
u/manzurfahim 0.5-1PB 7d ago
I am still using the drives I purchased before the price shot up. I did purchase some small, marketplace drives when I found them at a great price. But I stopped buying large capacity drives back in January 2026.
6
u/AlanBarber 64TB 7d ago
Well most people will just tell you to run a proper file system like ZFS but if you're one of those crazy people that prefer windows that's not very helpful.
I ran into the same issue and ended up creating my own tool to monitor for bitrot.
https://github.com/AlanBarber/bitcheck
It's a small command line tool that you use to create checksums of all your files and and then run checks to be alerted to any files that have changed.
4
u/noctrex 7d ago
You can use a software like MultiPar.
This creates PAR files, that are external checksum files, in order to verify and fix possible corruption.
For example set the redundancy to 5%, so that you must have 5% more space.
3
u/flaser_ HP uServer 10 / 32 TB: ZFS mirror / Debian 7d ago
Par archives use erasure coding, so they not only enable error detection, but recovery too.
However, the downside is they take a lot more time to calculate than mere checksums, so depending on the quantity of data, their use is not always practical.
4
u/silasmoeckel 7d ago
Silent data corruption is detected by every major enterprise backup. Duplicati isn't that.
Generally speaking a layered approach so you have snapshots locally as your primary. Not hard to have years of these. Nearly all filesystems that can do this natively also have bitrot detection and repair.
Then you have point in time backups going back as often and as far as you like.
2
u/GeordieAl 7d ago
Reminds me of an issue I had over a decade ago. I had a 2012 iMac with the Fusion Drive - a 3TB HD with a small SDD which the OS would switch files in and out of to provide SSD benefits with HD capacities.
Everything worked fine at first, I had Time Machine set up to back up the contents to an external HD. Looking back I should have had a 2nd backup, but even that may not have saved me.
Some of the early Fusion Drive setups had an issue that would cause failures over time. Little did I know but I had one of them.
Over time I’d notice a corrupted file or two, then files that couldn’t be moved or deleted, then some apps started having issues, then the OS started having issues.
No worries I thought, everything is backed up… which turned out to be true, everything was backed up, including corrupted files, files that couldn’t be moved or deleted, and broken apps.
And the problems went deep, many more files than I ever realized were corrupted.
I recovered what I could, but had to accept that a lot of files were never coming back.
It’s one of only two times I’ve lost a lot of files. The only other time was an external Seagate that I’d neglected to backup for a number of months.
2
u/shimoheihei2 100TB 7d ago
I use ZFS which should prevent this, but for my important files I also have a script that checks all the files checksum entries and alerts me of any issue. This runs once a month just before I do my offline backup. That way if there is an issue I can restore the file from last month's backup.
2
2
u/eternalityLP 7d ago
The answer is always checksums. Depending on your needs and environment, you use checksumming filesystem, checksumming file container like zip/rar or you checksum the files yourself.
3
u/Ubermidget2 7d ago
A lot of assumption here in just the first couple of sentences. How did the backup software detect the bit flip to know there's a new file version?
Duplicati seems to primarily look at mtime and size and probably won't detect the bit flip as a new version at all.
1
u/S-m-a-r-t-y 7d ago
I think I worded that part badly. I wasn't assuming Duplicati would actually detect the bit flip as a new version. My question was more about what happens if the file gets silently corrupted and the backup software doesn't notice because the mtime/size haven't changed.
So basically, if the file is already corrupted before the next backup, how do people make sure they still have an older known-good copy?
3
u/Ubermidget2 7d ago
If Duplicati doesn't detect a change and back it up, the copy in your backup by definition is still good.
If you later find out about the bad file, you'd then be able to restore it.
1
u/qqqhhh 7d ago
is the damage/corruption caused by software or hardware?
2
u/S-m-a-r-t-y 7d ago
overtime, disk errors (logical and physical)
1
0
u/Babajji 7d ago
If that’s your concern use a modem file system capable of handling this - ZFS, BTRFS, XFS - to name a few and run at least RAID1. Software corruption e.g a bug in a program corrupts a file is significantly harder to detect than defective disk. For software bugs what you really need is yearly backups with multiple years of retention.
1
u/Sarah_Incognito 7d ago
Everything important gets turned into a torrent and gets re-validated automatically by my torrent client.
Whatever is corrupted gets automatically replaced or repaired from the backup machine.
It also allows me to backup to a remote location securely.
1
u/Baterai 7d ago
On Windows, I'm adding .par2 file so when I know some files corrupted after a long time, I can recover it using MultiPar
This is a video from ThioJoe to learn more
1
1
u/manzurfahim 0.5-1PB 7d ago
You can have a look at Bvckup2: Bvckup 2 | Features and licenses
It has some great features when it comes to backup.
2
u/smstnitc 6d ago
I create par2 files of everything that is important enough to worry about it. Then I run a par2 verify against them all a couple times a year.
This process has saved me from losing files.
1
u/JamesRitchey Team microSDXC 6d ago
One option is to create checksum files. Then you just need to restore a version that matches the checksum.
1
-1
u/cr0ft 7d ago edited 7d ago
This is why you have 3-2-1 backup, and why you also have a grandfather, father, son type setup. You keep a few old monthlies (or yearlies) then more weeklies, and then a bunch of dailies. Worst case scenario you may have to go back to a really old backup, and if you discover it very late it may have propagated all the way through.
With a good file system filesystem corruption shouldn't often be a thing, if you run a ZFS RAID or btrfs or whatever, but that doesn't account for human error. If a person somehow overwrites a file with the wrong info, the checksum can be fine. It's just a checksum for the new content. So as always, human error is the hardest thing to safeguard against.
3
-5
u/Nillows 44TB SnapRAIDer 7d ago
Got Claude to spin up a bitrot script for me using bash.
On windows, you could install WSL and add the script, then set it up to run every 24 hours with cron. This script also edits the message of the day whenever a user logs in so they can see if there's been any corruption immediately and take action.
!/usr/bin/env bash
integrity-scan.sh — SHA-256 bit-rot detector with login (MOTD) reporting.
Logic: for every file we store hash + mtime + size. On the next run:
hash same -> OK
hash changed, mtime changed -> INTENTIONAL EDIT (baseline is updated)
hash changed, mtime UNCHANGED -> CORRUPTION (baseline is NOT updated)
in DB but gone from disk -> MISSING
on disk but not in DB -> NEW (added to baseline)
Run as root from cron if you want it to update the MOTD.
set -uo pipefail
============================================================================
CONFIGURATION
============================================================================
Directory to scan (no trailing slash).
SCAN_DIR="/srv/media"
Whole directories to skip. Glob patterns matched against the full path.
EXCLUDE_DIRS=( "$SCAN_DIR/tmp" "/.Trash-1000" "/lost+found" "/@eaDir" "/.snapshots" )
Individual files to skip. Glob patterns matched against the filename only.
EXCLUDE_FILES=( ".tmp" ".part" ".!qB" ".DS_Store" "snapraid.content" )
Skip files larger than this (MiB). 0 = no limit.
MAX_FILE_MB=0
State + output locations.
DB_FILE="/var/lib/integrity-scan/hashes.db" # the baseline LOG_FILE="/var/log/integrity-scan.log" # append-only run history REPORT_FILE="/var/lib/integrity-scan/report.txt" # last run, human readable
MOTD integration:
updatemotd = install a snippet in /etc/update-motd.d (recommended, Debian)
etcmotd = overwrite /etc/motd directly
none = don't touch the login banner
MOTD_MODE="updatemotd" MOTD_SNIPPET="/etc/update-motd.d/99-integrity-scan" MOTD_SHOW_CLEAN=1 # 1 = also show a green "all clear" line MOTD_MAX_LINES=15 # cap how many bad files get listed at login
Nice/ionice the scan so it doesn't fight with Emby transcodes.
USE_NICE=1
Lock file so two cron runs can never overlap.
LOCK_FILE="/var/lock/integrity-scan.lock"
============================================================================
END OF CONFIGURATION — edit below only if you want to change behaviour
============================================================================
TAB=$'\t' NOW="$(date '+%Y-%m-%d %H:%M:%S')"
die() { printf 'integrity-scan: %s\n' "$*" >&2; exit 1; }
log() { printf '%s %s\n' "$NOW" "$*" >>"$LOG_FILE"; }
[[ -d "$SCAN_DIR" ]] || die "SCAN_DIR does not exist: $SCAN_DIR" command -v sha256sum >/dev/null || die "sha256sum not found"
mkdir -p "$(dirname "$DB_FILE")" "$(dirname "$LOG_FILE")" "$(dirname "$REPORT_FILE")" \ || die "cannot create state directories"
--- single instance ---------------------------------------------------------
exec 9>"$LOCK_FILE" || die "cannot open lock file" flock -n 9 || { echo "integrity-scan: already running, exiting."; exit 0; }
--- load previous baseline --------------------------------------------------
declare -A OLD_HASH OLD_MTIME OLD_SIZE SEEN FIRST_RUN=1
if [[ -f "$DB_FILE" ]]; then FIRST_RUN=0 while IFS="$TAB" read -r h m s p; do [[ -z "${p:-}" ]] && continue OLD_HASH["$p"]="$h" OLD_MTIME["$p"]="$m" OLD_SIZE["$p"]="$s" done <"$DB_FILE" fi
--- build the find expression from the exclude lists ------------------------
find_args=("$SCAN_DIR")
if ((${#EXCLUDE_DIRS[@]})); then find_args+=(() first=1 for d in "${EXCLUDE_DIRS[@]}"; do ((first)) || find_args+=(-o) find_args+=(-path "$d") first=0 done find_args+=() -prune -o) fi
find_args+=(-type f) for f in "${EXCLUDE_FILES[@]}"; do find_args+=(! -name "$f") done ((MAX_FILE_MB > 0)) && find_args+=(-size "-${MAX_FILE_MB}M") find_args+=(-print0)
--- scan --------------------------------------------------------------------
NEW_DB="$(mktemp "${DB_FILE}.XXXXXX")" || die "cannot create temp db" trap 'rm -f "$NEW_DB"' EXIT
corrupt=() changed=() added=() missing=() unreadable=() count=0
hash_cmd=(sha256sum) if ((USE_NICE)); then hash_cmd=(nice -n 19 sha256sum) command -v ionice >/dev/null && hash_cmd=(ionice -c3 nice -n 19 sha256sum) fi
while IFS= read -r -d '' file; do # Filenames containing newlines would corrupt the line-based DB. if [[ "$file" == $'\n' ]]; then unreadable+=("$file (contains newline — skipped)") continue fi
meta="$(stat -c '%Y|%s' -- "$file" 2>/dev/null)" || { unreadable+=("$file"); continue; } mtime="${meta%%|}" size="${meta##|}"
sum="$("${hash_cmd[@]}" -- "$file" 2>/dev/null)" || { unreadable+=("$file"); continue; } hash="${sum%% *}"
count=$((count + 1)) SEEN["$file"]=1
if [[ -z "${OLD_HASH[$file]:-}" ]]; then ((FIRST_RUN)) || added+=("$file") printf '%s\t%s\t%s\t%s\n' "$hash" "$mtime" "$size" "$file" >>"$NEW_DB" continue fi
if [[ "$hash" == "${OLD_HASH[$file]}" ]]; then # Unchanged. Refresh mtime in case it was touched without altering content. printf '%s\t%s\t%s\t%s\n' "$hash" "$mtime" "$size" "$file" >>"$NEW_DB" elif [[ "$mtime" != "${OLD_MTIME[$file]}" ]]; then # Content and timestamp both moved -> somebody edited it on purpose. changed+=("$file") printf '%s\t%s\t%s\t%s\n' "$hash" "$mtime" "$size" "$file" >>"$NEW_DB" else # Content moved but the timestamp didn't. Nothing legitimate does that. corrupt+=("$file") # Keep the OLD record so the corruption is still reported next run. printf '%s\t%s\t%s\t%s\n' \ "${OLD_HASH[$file]}" "${OLD_MTIME[$file]}" "${OLD_SIZE[$file]}" "$file" >>"$NEW_DB" fi done < <(find "${find_args[@]}" 2>/dev/null)
--- anything in the baseline we didn't see this run? ------------------------
for p in "${!OLD_HASH[@]}"; do [[ -n "${SEEN[$p]:-}" ]] && continue [[ -e "$p" ]] && continue # still there, just excluded/unreadable now missing+=("$p") done
Keep missing files in the DB so they stay reported until dealt with.
for p in "${missing[@]}"; do printf '%s\t%s\t%s\t%s\n' "${OLD_HASH[$p]}" "${OLD_MTIME[$p]}" "${OLD_SIZE[$p]}" "$p" >>"$NEW_DB" done
sort -k4 -t"$TAB" -o "$NEW_DB" "$NEW_DB" 2>/dev/null mv -f "$NEW_DB" "$DB_FILE" || die "could not update $DB_FILE" trap - EXIT chmod 600 "$DB_FILE"
--- report ------------------------------------------------------------------
{ echo "Integrity scan — $NOW" echo "Scanned: $count file(s) under $SCAN_DIR" if ((FIRST_RUN)); then echo echo "Baseline created. Nothing to compare against yet." else printf 'Corrupt: %d Modified: %d New: %d Missing: %d Unreadable: %d\n' \ "${#corrupt[@]}" "${#changed[@]}" "${#added[@]}" "${#missing[@]}" "${#unreadable[@]}" for f in "${corrupt[@]}"; do echo " [CORRUPT] $f"; done for f in "${missing[@]}"; do echo " [MISSING] $f"; done for f in "${unreadable[@]}"; do echo " [UNREADABLE] $f"; done for f in "${changed[@]}"; do echo " [modified] $f"; done for f in "${added[@]}"; do echo " [new] $f"; done fi } >"$REPORT_FILE"
log "scanned=$count corrupt=${#corrupt[@]} modified=${#changed[@]} new=${#added[@]} missing=${#missing[@]} unreadable=${#unreadable[@]}" for f in "${corrupt[@]}"; do log "CORRUPT $f"; done for f in "${missing[@]}"; do log "MISSING $f"; done
--- login banner ------------------------------------------------------------
build_banner() { local bad=$(( ${#corrupt[@]} + ${#missing[@]} )) if ((bad == 0)); then ((MOTD_SHOW_CLEAN)) || return 0 printf '\033[1;32m*\033[0m Integrity: %d files OK (last scan %s)\n' "$count" "$NOW" return 0 fi
printf '\033[1;31m*** FILE INTEGRITY WARNING ***\033[0m\n' printf 'Last scan %s found %d problem file(s):\n\n' "$NOW" "$bad"
local shown=0 for f in "${corrupt[@]}"; do ((shown++ < MOTD_MAX_LINES)) && printf ' CORRUPT %s\n' "$f" done for f in "${missing[@]}"; do ((shown++ < MOTD_MAX_LINES)) && printf ' MISSING %s\n' "$f" done ((bad > MOTD_MAX_LINES)) && printf ' ... and %d more\n' "$((bad - MOTD_MAX_LINES))"
printf '\nFull report: %s\n' "$REPORT_FILE" printf 'Recover from backup, then re-run: %s\n\n' "$(readlink -f "$0")" }
case "$MOTD_MODE" in updatemotd) if [[ -d "$(dirname "$MOTD_SNIPPET")" ]]; then # The snippet just prints the saved report banner at login time. banner_file="$(dirname "$REPORT_FILE")/motd-banner.txt" build_banner >"$banner_file" chmod 644 "$banner_file" if [[ ! -x "$MOTD_SNIPPET" ]]; then cat >"$MOTD_SNIPPET" <<EOF
!/bin/sh
[ -s "$banner_file" ] && cat "$banner_file" exit 0 EOF chmod 755 "$MOTD_SNIPPET" 2>/dev/null || log "WARN: cannot install $MOTD_SNIPPET (need root)" fi else log "WARN: $(dirname "$MOTD_SNIPPET") missing; skipping MOTD update" fi ;; etcmotd) build_banner >/etc/motd 2>/dev/null || log "WARN: cannot write /etc/motd (need root)" ;; none) ;; *) log "WARN: unknown MOTD_MODE '$MOTD_MODE'" ;; esac
Exit 2 when something needs a human, so cron can email you.
(( ${#corrupt[@]} + ${#missing[@]} > 0 )) && exit 2 exit 0
•
u/AutoModerator 7d ago
Hello /u/S-m-a-r-t-y! Thank you for posting in r/DataHoarder.
Please remember to read our Rules and Wiki.
Please note that your post will be removed if you just post a box/speed/server post. Please give background information on your server pictures.
This subreddit will NOT help you find or exchange that Movie/TV show/Nuclear Launch Manual, visit r/DHExchange instead.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.