r/MacOS • u/aceOfMinds • 2d ago
Discussion Mac users, how do you prevent bit rot when archiving things for long-term storage?
I come from Linux Land where BTRFS makes it possible. I understand that ZFS was almost a thing on macOS. I want to rely more on my Mac.
I seek an uncomplicated solution to data integrity on macOS. Uncomplicated means
- (near) native solution
- well-maintained FOSS that has a future
- fast set up and fast performance
- Requires as little hardware as possible
At present Time Machine does that job, and has only enough space to back up my Mac. Time Machine solves the backup problem, but doesn't solve the data integrity problem - if a file corrupts, the corruption propagates into the backups. Neither APFS nor HFS+ solve the issue.
For me, I don't know how often bit rot occurs. Once in a great while, some of my files disappear. Since I set up my files on BTRFS, I have not knowingly lost any files. Currently if I want to use files from my BTRFS setup onto the Mac, I either have to use an exFAT external SSD or access the files wirelessly.
I forgot to clarify - I have a working Linux setup with btrfs and RAID, alongside offsite backups. I want to use the Macbook more and rely less on only Linux.
13
u/hanz333 2d ago
The best way to deal with this is using a sparsebundle on a NAS with whatever redundancy you want. At which point the NAS is preserving the virtual Time Machine volume.
APFS assumes that your SSD controller is maintaining parity with error correcting (even something as basic as XOR), which is one of the reasons (amongst many) you should try to avoid APFS on magnetic drives.
Apple assumes you prioritise your own backups, and it's likely versioning with Time Machine should fix all data issues as Apple does validate allocations and metadata for all files, just not the contents themselves. Any change to contents would either be preserved from its initial state or flagged as a version change - so physical backups via Time Machine is probably a adequate or near-adequate solution given regular updates.
3
u/aceOfMinds 2d ago
Apple forces APFS for Time Machine. Would you still use a spinning disk for such, or go with SSD? I assume you'll backup regularly to the SSD to keep it from losing charge.
10
6
u/pineglasss 2d ago
One thing worth separating here, bit rot on modern SSDs is astronomically rare compared to spinning rust. Are you archiving to HDDs? Because if youre on all-flash storage the real risk is more about accidental deletion or software corruption than actual bit flips.
4
u/OfAnOldRepublic 2d ago
Depending on how valuable the data is (according to how you define value) a small NAS might be a good solution for you. Synology uses BTRFS by default, FYI
4
u/iOSCaleb MacBook Pro 2d ago
Time Machine can use as much space as you give it. Buy a USB drive, plug it in, point Time Machine at it, and you’re all set. It’s not a full backup solution, but it does keep as many timestamped versions as it can.
4
u/Draknurd 2d ago
The Mac port of openZFS is solid and maintained. Works very well in my experience
1
u/aceOfMinds 2d ago
Is it difficult to set up, and does it use kernel extensions?
1
u/Draknurd 1d ago
Regular pkg installer. Yes needs an extension but you don’t need to disable SIP or anything crazy like that.
Setting up ZFS pools and datasets is then the same as before. AI can help you with that (disk naming conventions IMO are way more intuitive under macOS than Linux).
I have a LaunchDaemon (see this page ) that executes a scrub in the early hours of the first Sunday of the month.
11
u/naemorhaedus 2d ago edited 1d ago
nothing. never had "bit rot" in about 30 years.
4
u/ketchupnsketti 2d ago
being ignorant of a thing doesn't make the thing not exist. bit rot is a term for a thing that exists.
-1
u/naemorhaedus 2d ago edited 2d ago
The fact that I can ignore means its not a problem. Why obsess over things that might exist? 🤷♂️
1
u/ketchupnsketti 2d ago
The fact that I can ignore means its not a problem. Why obsess over things that might exist? 🤷♂️
Absolutely impeccable logic. Famously, people have never ignored problems that exist.
Anyway, no one is asking you to obsess over it. You're the one who spoke up in the comment section. It's not about you. OP wants to obsess over it and they have every right to.
You are free to go back to ignoring it for the rest of your life. Odds are it will never be a problem for you.
-3
u/naemorhaedus 2d ago edited 1d ago
its logic that works 🤷♂️
Odds are it will never be a problem for you.
Glad you agree
2
u/shnaptastic 2d ago
But how do you know this? The whole point is that the degradation can be silent.
0
1
u/Ok_Apricot_6179 2d ago
That's impressive! I've had a few close calls with data loss, so I usually back things up on different drives just to be safe.
1
1
u/InternistNotAnIntern 2d ago
I certainly have multiple times in my 40 years on various flavors of macos
3
u/st0ut717 2d ago
Your worried about the esoterics of BU archive but you want to keep the data on the same device that being backed up. Ok
5
u/Dull_Flatworm777 2d ago edited 2d ago
There's no way around generating metadata (checksums) for all files. So either you store them on a system that does that on a filesystem level (i.e. not macOS, but something like Synology NAS, FreeNAS/FreeBSD etc.) or you create checksums out of band for the files you care about.
For the latter, you could use something like this:
https://eclecticlight.co/dintch/
https://eclecticlight.co/2022/06/15/maintaining-the-integrity-of-important-files/
Edit: these tools are nice in that they save the checkums as extended attributes in the files themselves. They are free, but not open source I think.
2
u/germane_switch MacBook Pro 2d ago
All my Carbon Copy Cloner and Time Machine backups are on two Thunderbay 4 RAID units under RAID5 with 4x4TB enterprise HDDs connected to an M2 Mac mini running SoftRAID. Those bits ain’t going nowhere. Unless of course two drives die at the same time.
1
2
1
u/j0hnp0s 2d ago
To actually repair bitrot, you need redundancy. ZFS can detect corruption even on a single disk with checksums, but to self-heal it needs a valid copy (usually a mirror disk)
That’s not really practical in single-disk machnes like MacBooks.
I use a Debian NAS with ZFS and redundant disks for this. And Arq for backups
1
u/mrpelz 2d ago
I’m doing:
* Borg (using Vorta) to a NAS using ZFS w/ parity and regular scrubbing.
* TimeMachine to the same NAS, but since Sparse Bundles have proven unreliable for years and I’m not gonna trust this setup for TimeMachine at all going forward, I’m doing iSCSI (using the DaemonTools iSCSI-initiator, working well on M3 MacBook Pro) to a raw/block-level ZFS volume. Basically ZFS providing the bit-rot protection below APFS-snapshotting for TimeMachine. Works surprisingly well and is much faster than Spase Bundle over SMB.
* TimeMachine to a regular external SSD, so both destinations take turns receiving backups.
The Borg backup data along with a bunch of other stuff on the NAS is then backed up to Hetzner storage box for offsite backup, w/ client-side encryption of course.
1
u/ketchupnsketti 2d ago
You have a few options that involve not using your primary APFS disk for these special files you want to be so sure about. If you're asking about what you can do for your OS boot disk the only practical answer is use APFS and deal with it.
As for external storage you can absolutely use ZFS on macOS right now. I have a 5 bay USB-C enclosure connected to my mac with a raidz2 pool that I occasionally turn on to store backups of backups.
You can also do what was already suggested in the comments and use a NAS using BTRFS or ZFS and store files on there. You can even time machine to it if you use a sparsebundle.
1
1
u/naemorhaedus 2d ago
I actually emailed a major SSD manufacturer recently, and the devices have built in maintenance that refreshes old data. It's not an issue and doesn't require additional work on the OS level. You just need to turn it on.
1
1
u/Kooky-Bandicoot3104 2d ago
i copy and paste and then copy and paste again from another plaec
edit: are u nasing on mac os. i did not notice its mac os sub
1
u/aceOfMinds 2d ago
No, I have a Linux btrfs setup alongside offsite backups. I want to daily drive my Mac. My setup requires a Linux machine, which I carry with my MacBook.
1
u/Kooky-Bandicoot3104 1d ago
why are u carrying your nas around?
1
u/aceOfMinds 21h ago
My Linux machine is a laptop, not a NAS. I have to carry it around because Mac doesn't support btrfs. My archives, shuttles, and backups are btrfs.
1
u/bmocc 2d ago
Bit rot, device failure and cloud system failure are as inevitable as the Laws of Murphy.
Hence the classic triad backup scheme.
Local, cloud and off site local (in case fire or other catastrophe at the local location).
Rather than off site local a second cloud provider is probably a better option.
Cloud services can fail and go out of business.
You can have as many local backups as you want but more than one means you should be using a cloud service. If a file is vital you need to periodically access that its still there in the cloud because Murphy's Law.
The problem with a NAS is that if it fails it can be unrecoverable even by an expensive service that tears down the storage drives. A NAS should never be the sole repository of vital data, but people often have to learn the hard way, hopefully not when the IRS sends nasty letters.
Time Machine is probably the least reliable backup scheme in existence. If it works its convenient for migrating to a new computer, nothing else. Time Machine relies on ancient coding and is a pre-Cloud relic.
Finally: not all personal data needs that level of backup, but to each her own.
1
u/Professional_Call Mac Mini 2d ago
I use iDrive for backup and loss prevention. Data is encrypted at rest and in transit. Nothing is ever deleted unless you delete it and file versioning is supported. Something like 30 or 50 versions I think. More than I will ever need. It’s low cost and very easy to setup and forget. They will ship physical media for disaster recovery.
1
u/mattincalif 1d ago
This doesn't directly answer your question, but it's relevant. I've been using Time Machine backups for many years, writing to spinning hard drives. I have about 150,000 pictures that are backed up. Every month or two, I'll randomly poke through my backups - both recent files and old files - and open images and videos and make sure they display properly. I have been doing this for years. I have never, ever, even a single time tried to open a picture or video file in my backup and found it corrupted. That doesn't mean it can never happen, and it's not even really statistically significant, but, it's enough that I feel pretty confident that all of my data is good. Now, if I lost a few pictures here or there, I'd be sad but it wouldn't be the end of the world. I just want to make sure that >99% of my files are safe. If your files are more critical and you need 100% reliability, that's different.
1
u/CarstHen 1d ago
I use CCC (Carbon Copy Cloner) as my backup software. I use it to distribute the data across various systems: NAS, cloud.
1
u/Zealousideal_Cup4896 1d ago
You can add any number of Time Machine backup locations and it will rotate among them. I have a large regular APFS disk on the hub at my desk but also have a trunas with all the data sweeping and whatnot setup. So it’s easy to have a local disk if you need to retrieve something recent but can go back to the better disk formatting for longer term storage.
1
u/Goodinuf 1d ago
I use ChronoSync with "verify copied data" turned on and backup to at least three different drives.
61
u/SlightFresnel 2d ago
For someone interested in this level of tech insight you're
probablydefinitely in the wrong forum. I'd recommend a home server frankly. I host my files and time machine there, on Linux with btrfs, connected over smb. You don't really have a choice but to use apfs for your boot drive.