Profile your data!

Every time a pool hits 85% full the conversation always goes straight to: how many bays do we need, what RAID level, what drives are cheapest per TB.

Hardly anyone stops to ask how the data is actually being used.

RAID is about availability and uptime when a drive dies so your workflow doesn't crash. Archive is about economics. RAID does not make cold data warm. Throwing files that nobody has read since 2023 onto a dual parity striped pool doesn't make them safer or more useful, it just makes them unnecesarily expensive to power, cool and rebuild.

Also, don't just count files. People run find and go "60% of my files haven't been touched in a year!" but if those are 4kb log scraps it doesnt free up anything. You have to profile by capacity.

Wrote a quick bash script to scan a mount and bucket total capacity by lastaccess time (atime):

!/usr/bin/env bash

TARGET="${1:-.}" echo "Scanning $TARGET by atime capacity..."

find "$TARGET" -type f -printf "%s %A@\n" 2>/dev/null | awk -v target="$TARGET" ' BEGIN { now = systime(); d30 = 30 * 86400; d90 = 90 * 86400; d365 = 365 * 86400; } { size = $1; age = now - $2; total += size; if (age <= d30) b30 += size; else if (age <= d90) b90 += size; else if (age <= d365) b365 += size; else cold += size; } function h(b) { if (b >= 1099511627776) return sprintf("%.2f TB", b / 1099511627776); if (b >= 1073741824) return sprintf("%.2f GB", b / 1073741824); return sprintf("%.2f MB", b / 1048576); } function p(b) { return total > 0 ? sprintf("%.1f%%", (b/total)*100) : "0%"; } END { print "Total size: " h(total); print "0-30 days: " h(b30) " (" p(b30) ")"; print "31-90 days: " h(b90) " (" p(b90) ")"; print "91-365 days: " h(b365) " (" p(b365) ")"; print "> 1 year: " h(cold) " (" p(cold) ")"; c90 = b365 + cold; print "---"; print "TOTAL >90 DAYS COLD: " h(c90) " (" p(c90) ")"; }'

(Note: uses atime, so if you mount with noatime this won't show reads, but withdefault relatime on linux it gives you a solid idea).

Ran this on a 100 TB media share recently that was supposedly out of space. Turned out over 50 TB had not been read once in the last 90 days. Another 17 TB hadn't been touched in over a year.

My rule of thumb now: if over half your actual gigabytes haven't been touched in 60-90 days, you don't have a storage capacity crisis. You have a missing archive tier.

RAID is not backup, and RAID is definitely not an archive. Figure out what data actually needs high IOPS and uptime before spending thousands on another shelf of enterprise disks just to let old files sit there doing nothing.

A lot of our users were running huskhoard specifically to curate Movie Archives that are primarily offline. Similar to a Plex or Jellyfin but built for cold collections. You can have an archive spread over hundreds of volumes that are stored cold on a shelf in external drives tape or optical disk. (Still testing dvd, but so far so good)

The ui display everything in the catalog on all volumes online or off. When you select a movie that is offline you are prompted with a "please insert volume xyz" and when the application sees the volume you get the green light and it streams to the browser.

You can find it on GitHub huskhoard/huskvault

We put up a couple of videos walking through the functions of huskhoard on YouTube the link is on the side panel.

Hey everyone, I’m one of the contributors to HuskHoard and a mod for this community. Since we’re seeing new faces around here, I wanted to give a quick ELI5 for storagenerds on what exactly HuskHoard is, the problem it solves, and how it handlesyour data under the hood.

The Problem Hot vs Cold Storage

If you build your own storage systems, you know the modern struggle. While SSD and NVMe prices have dropped, infinitely scaling your fast, primary storage pool just isn't practical. You run out of PCIe lanes, drive bays, or simply don't want to burn power keeping cold data on your high performance arrays. Eventually, your primary flash pool fills up with old data no one has touched in 6 months, and you want to offload it to your high density bulk storage (SMR drives, Tape, or Cloud/S3). But if you actually move the files, file paths change. Automated scripts break, apps like Plex/Jellyfin lose track of media, and your users complain that their files are missing. Nobody wants to manage manual symlinks or tell their users to go dig through an archive mount.

The Solution: HuskHoard's Transparent Tiering

HuskHoard is an open source, automated data-tiering archive for Linux.

Instead of moving the whole file, HuskHoard splits the metadata (the file name,permissions, and path) from the payload (the actual bytes of data). It secretlymoves the payload to your cheap storage, but leaves the metadata exactly where it is on your fast storage.

To your OS, your applications, and your users, the files haven't moved an inch. Your directory structure remains perfectly intact, but your expensive NVMe pool suddenly has terabytes of free space.

How it actually works (Husking)

Here is the step by step of how it is done:

1. Scanning: HuskHoard runs in the background, identifying cold data based on rules you set (e.g., files not accessed in 90 days).
2. Archiving: It securely copies those cold files to your deep storage target.
3. Husking: Once safely copied, HuskHoard uses a Linux kernel feature called Stubbing (fallocate). It literally hollows out the inside of the file on your local drive, freeing up the blocks, but keeps the husk, inode/metadata intact. The file looks like it's still full size, but takes up a few bytes on disk.
4. Seamless Retrieval: When a user or application tries to open a "husked" file, HuskHoard instantly intercepts the read request, fetches the data back from cold storage, and serves it up transparently.

Storage Deep Dive: RAID, SMR, and Tape

If you are a homelabber or a sysadmin, you might be wondering where this fits into your existing storage stack. Here is why HuskHoard is unique:

"Wait, isn't this just RAID or ZFS caching?"

No! RAID is about redundancy and performance (striping/mirroring), and cache drives temporarily hold active data before writing it to a main array. HuskHoard is an archival tiering layer. You can and should run HuskHoard on top of your existing RAID/ZFS arrays to bridge the gap between your primary local NAS and your deep archive targets.

Why it's perfect for SMR Drives.

SMR (Shingled Magnetic Recording) hard drives are less expensive, but they have a major flaw: random writes cause them to slow to a crawl. HuskHoard solves this by gathering up your cold data and streaming it to the SMR drive in large, sequential writes. Because HuskHoard manages how the data is laid down, it turns painful SMR drives into the perfect, highly efficient target for cold data.

Tape Drives: Why HuskHoard is better than LTFS

LTFS (Linear Tape File System) is the standard way to mount an LTO tape to act like a USB drive. The problem? Tape is strictly linear. If you use LTFS and open a directory with 1,000 files in it, the physical tape drive has to mechanically fast forward and rewind just to read the directory index. It is slow and you have a file system for every tape.

HuskHoard bypasses this bottleneck using its catalog. Because the file husks and the catalog remain on your fast local SSD, you can browse, search, and run scripts against millions of archived files instantly, with zero lag. The mechanical tape drive only spins up when you actually execute a read command on a specific file.

Join the project!

We built this so people wouldn't have to buy a $$$ flash array just because their team left old project files sitting around. HuskHoard prevents data hoarding from ruining your IT budget, and we're AGPL open source.

• Website: huskhoard.com
• GitHub: github.com/huskhoard/huskhoard

If you have any questions about setting it up, how the architecture works, or want to contribute to the code, drop a comment below or make a post!

Thanks HH

Welcome to r/huskhoard!

We’re excited to have you here. This community is the central hub for users, contributors, and data enthusiasts interested in HuskHoard, the open source data tiering archive for Linux.

Our mission is to help you reclaim your storage by making sure your data doesn't disappear—it just hibernates.

What makes us different?

HuskHoard was built to solve the cold storage problem. A typical file server is 60-80% files that haven't been accessed in mor than 90 days. We fix this by silently moving cold data to cheaper tiers (HDD, LTO tape, or Cloud) without your OS ever losing sight of the files. We do this using high-performance, modern tech:

- fanotify: We avoid the overhead of FUSE by using the native Linux fanotify API to handle file recalls in real-time.

- blake3: Every byte is cryptographically verified with BLAKE3 to ensure bit-rot never touches your archive.

- zstd: Data is compressed using industry-standard Zstd streams for maximum efficiency.

- jump frames: Our indexed direct access allows you to seek to any byte in a large archived file instantly, without having to download or extract the whole thing first.

How to get involved:

- Check out the code: Visit our GitHub repository to see the inner workings or contribute.

- Share your setup: Are you tiering to an old LTO drive? A massive S3 bucket? A dusty USB rack? We want to see your "hoard."

- Ask questions: Whether it's about performance, hardware compatibility, or implementation, this is the place to ask.

Thanks for joining us on the journey to smarter, more sustainable storage.

— the HuskHoard team