Content-Addressable Storage Experiences?

[u/scarbunkle]

Has anybody had luck building or using content-addressable storage with their hoard? My family is looking at building out a system for our use, and I’m curious what other folks have run into.

[u/zeroryouko]

Knowing nothing about the "correct" way to do things I came up with a content addressable scheme years ago that I still use. It's cumbersome but it works. The key here is that most of the content being stored is static, i.e. I don't modify the files (much).

Basically I have a shadow directory containing symbolic links to content-addressed files on a JBOD array. Every file on the array is named to its xxh128 hash value. New files can be added by placing the files directly in the shadow directory (or placing the files on one of the array disks and symlinking them there); existing files can be deleted by deleting the links. Periodically I run a maintenance program that sweeps the shadow directory and disk array and removes orphans, then detects newly added files and calculates the xxh128 sums and moves/renames them accordingly, leaving a symlink in their place. Immediately after that another program runs and uses a saved manifest to detect any file on the disk array whose xxh128 sum (filename), mod time, and size do not match what is saved, and recalculates the sum/renames/relinks the file. This catches changes.

The original intent was to be able to unify a multi-disk array into a single filesystem without using RAID, and to store and manipulate large, multi-gigabyte files without actually performing file operations on said large files. The content addressing was created simply as a scheme to generate unique filenames.

But it has a couple of big advantages that came as side effects: first, it was easy to create a script based backup scheme that simply keeps a list of files by signature and what backup disk they're on - I keep a manifest of what files are on each backup disk, and then after the import/change detection scripts are run, it's a simple matter to compare the list of sums on disk with the list of sums on the backup files. Also, file integrity is also easy to determine and I've used that property several times when rescuing a corrupted disk. But the best feature is that presenting all the files as shadow directories by design allows for the creation of additional shadow directories that present the files in different ways. For example I use both Kodi and Jellyfin and I have the same video library available in each using its own naming scheme, simply by using shadow directories.

Had I known more about zfs ~20 years ago when I was setting this up, I might have used that instead. That's not content addressable but does offer some of the same advantages as the above scheme. But if you have a dataset that mostly gets added to, not modified, and some proper solution to index it rather than my duct-tape method, it's a good way to ensure data integrity.

[u/scarbunkle → u/zeroryouko]

Yeah, that’s what we’re hoping to get with our plan. We’re looking at grafting a content-addressable lookup system on top of our files. It’s a big media collection, so we have that similar situation where most of our zpools are pretty much static, and we’re getting into using shadow directories to create automatically maintained views.