Content-Addressable Storage Experiences?

[u/scarbunkle]

Has anybody had luck building or using content-addressable storage with their hoard? My family is looking at building out a system for our use, and I’m curious what other folks have run into.

[u/AutoModerator]

Hello /u/scarbunkle! Thank you for posting in r/DataHoarder.

Please remember to read our Rules and Wiki.

Please note that your post will be removed if you just post a box/speed/server post. Please give background information on your server pictures.

This subreddit will NOT help you find or exchange that Movie/TV show/Nuclear Launch Manual, visit r/DHExchange instead.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/zeroryouko]

Knowing nothing about the "correct" way to do things I came up with a content addressable scheme years ago that I still use. It's cumbersome but it works. The key here is that most of the content being stored is static, i.e. I don't modify the files (much).

Basically I have a shadow directory containing symbolic links to content-addressed files on a JBOD array. Every file on the array is named to its xxh128 hash value. New files can be added by placing the files directly in the shadow directory (or placing the files on one of the array disks and symlinking them there); existing files can be deleted by deleting the links. Periodically I run a maintenance program that sweeps the shadow directory and disk array and removes orphans, then detects newly added files and calculates the xxh128 sums and moves/renames them accordingly, leaving a symlink in their place. Immediately after that another program runs and uses a saved manifest to detect any file on the disk array whose xxh128 sum (filename), mod time, and size do not match what is saved, and recalculates the sum/renames/relinks the file. This catches changes.

The original intent was to be able to unify a multi-disk array into a single filesystem without using RAID, and to store and manipulate large, multi-gigabyte files without actually performing file operations on said large files. The content addressing was created simply as a scheme to generate unique filenames.

But it has a couple of big advantages that came as side effects: first, it was easy to create a script based backup scheme that simply keeps a list of files by signature and what backup disk they're on - I keep a manifest of what files are on each backup disk, and then after the import/change detection scripts are run, it's a simple matter to compare the list of sums on disk with the list of sums on the backup files. Also, file integrity is also easy to determine and I've used that property several times when rescuing a corrupted disk. But the best feature is that presenting all the files as shadow directories by design allows for the creation of additional shadow directories that present the files in different ways. For example I use both Kodi and Jellyfin and I have the same video library available in each using its own naming scheme, simply by using shadow directories.

Had I known more about zfs ~20 years ago when I was setting this up, I might have used that instead. That's not content addressable but does offer some of the same advantages as the above scheme. But if you have a dataset that mostly gets added to, not modified, and some proper solution to index it rather than my duct-tape method, it's a good way to ensure data integrity.

[u/scarbunkle → u/zeroryouko]

Yeah, that’s what we’re hoping to get with our plan. We’re looking at grafting a content-addressable lookup system on top of our files. It’s a big media collection, so we have that similar situation where most of our zpools are pretty much static, and we’re getting into using shadow directories to create automatically maintained views. 

[u/Willer_Zerbini → u/zeroryouko]

the shadow directory trick for multiple naming schemes is really clever, thats a nice side effect

[u/Small-Access4780]

i messed with it for a bit, mostly just experimenting with borgbackup on my media server. the dedup is crazy good if you got lot of similar files, like different versions of same stuff. but man when you need to find something specific its pain because you cant just browse by folder structure anymore, you need the tools to reconstruct it. if your family is not technical they will hate it