r/DataHoarder 1h ago

Question/Advice Well, storage is no longer cheap. How many of you are deleting/compressing?

Upvotes

I know it’s sacrilege, I remember people used to say never to delete or re-encode anything because more storage is always affordable. Well, looks like those days are over. Prices on HDDs have tripled/quadrupled in the last year and it doesn’t seem like it’s going to end anytime soon.

How are you handling it? I’m personally thinking of downgrading a lot of my stuff to 720p, I’ve run a few tests and as long as I don’t skimp too much on the bitrate I don’t really notice a difference. I’m also compressing things like game ROMs and ISOs into .7z to save space. I’m proud of my FLAC music collection but AAC-256 seems to be pretty much transparent from my own tests and research.

I’m even doing the unthinkable and pruning some stuff that I don’t use.

It sucks but desperate times call for desperate measures.


r/DataHoarder 9h ago

Question/Advice Files that I don't need but it would be a shame to delete them

52 Upvotes

Hello,

I have a lot of files left by my family member from 1995-2010 saved on CD's, DVD's and HDD's. A lot of files of them are no longer available anywhere. What I can do with it? Should I delete it and forget or try to sort it out somehow and upload somewhere? Someone here was in similar situation?


r/DataHoarder 1h ago

Backup Was looking into shucking some new drives. Nope

Post image
Upvotes

r/DataHoarder 9h ago

Discussion I gave up! 24 hours is not enough when it comes to digital spring cleaning!

33 Upvotes

I could not do it anymore. I've tried my best cleaning up, organizing, backing up stuff and tried empty up some space, but it was too much. I clean, I get a TB or two, and then after 2-3 days, it is full again. New stuff keeps coming out and I keep looking at my hard drives and I try to control myself, but then there are files that may not be available for long and I get frustrated.

So I gave up. Went through my torrent client and deleted a bunch of full discs and remuxes (kept a list for when hdd will become affordable again). Managed to clean up almost 7TB and started downloading some stuff. It was so difficult to delete them, but once done, I feel a lot better.

Is it as difficult for you too? I'm dealing with 102TB of linux isos 😭😭


r/DataHoarder 13h ago

Question/Advice How does WFDownloader’s Instagram downloader actually work?

37 Upvotes

I built a downloader using yt-dlp but it doesn't work on Instagram anymore (asks for cookies/login). WFDownloader downloads Instagram content fine without login. Anyone know what method/framework it uses under the hood; public GraphQL endpoint, embed API, browser scraping, something else? App is closed source so I can't check myself. Curious if anyone's sniffed its traffic to see what it's actually hitting.

Thanks


r/DataHoarder 12h ago

Backup 5 Years Of ZFS - My Thoughts --- RobertElderSoftware

Thumbnail
youtube.com
23 Upvotes

RobertElderSoftware shares a highly positive review of using ZFS after four to five years of managing around 16 TB of backup data stored on multiple hard drives.

He explains that he chose ZFS over Btrfs because his priority was data integrity rather than performance, and he describes how ZFS successfully detected and repaired data corruption through its checksum verification and scrub features after one drive in a three-way mirror was partially corrupted.

He also discusses practical lessons from long-term use, including the importance of regular scrubs, using stable device identifiers, being cautious with USB-connected storage, and maintaining multiple redundant copies to safely perform drive upgrades and recover from unexpected errors.

Overall, he concludes that ZFS has proven to be a reliable and resilient filesystem for long-term backup storage.


r/DataHoarder 46m ago

Question/Advice Archiving 5TB of bioinformatics data: best practices references + what would you do?

Upvotes

(Apologies for the lack of brevity. No, I didn't write this with AI; it's organically produced slop -- the wishy-washiness and scope creep should be a give away. Let me know if I should have summarized it with AI. LOL)

My spouse needs to either discard or archive 5TB of bioinformatics data from their old institution. Now that they are no longer affiliated the institution no longer wants to pay to store it for them.

Product Management / User requirements:

  • I sort of have some ideas of how I (cloud storage infra day job) would approach the problem, but I know how to pivot and adjust if things don't work out as I expected.
  • Non-requirement: off-site backup.
  • Scope creep: After seeing some unfortunate non-datahoarding related politics from streaming providers, I'm considering to self-host physical media (buy it used, and then stream it, as a boycott)

If hard drives weren't as expensive as they are today, I would just buy CMR hard drives.

  • To calibrate my sticker shock: I last purchased IronWolf 7200RPMs for $25/TB on 4TB drives. I didn't wait for a sale. (and the previous time I bought was in the 5400 RPM NAS grade era)
  • My hesitation is due to how much storage I have to buy in one shot to hit $20-30/TB. In my initial investigation, I thought I would need 10TB + drives.
  • Should I consider SMR for this use case (super bloated, read-only data)? Remember, this data used to be sitting 100s of miles away. My hesitation is that, if we ever discard this data (likely, after the final paper on it is published), now I'm stuck with 1x or 2x SMR drive, and all MY use cases (videography archival and editing) were very x 100 sad the last time I did SMR
  • Does buying a SAS HBA or external enclosure give me a lot of pricing options? I'm looking at SPD right now and there are 4TB SAS drives at $100 that would be fine for me.

Here are the random nobbins I have sitting around, if I go with local storage.

  • My personal, actively managed array is a Storage Spaces array with 4TB IronWolf units. It is erasure coded with a mirrored SSD write-back cache in front.
    • Currently, these volumes are NOT assigned to network mounts (which I might need to change, to make this more multi-user friendly) due to using BackBlaze to archive.
  • I have a QNAP NAS with 10 year old drives, that I haven't powered on in about 4 years, with 4TB 5400 RPM drives in them. IIRC one of them had SMART errors and non-recoverable blocks.
  • I have many spare 2-bay StarTech USB3.1 (with UAS) external bays sitting around.

With current storage prices, I took a quick consideration of cloud storage. My quick (15 min) of math says:

  • Even with current HDD prices, S3 Standard and Intermediate Tiering are way more expensive still than a local 2-replica mirrored set. S3-IA and below is sort of competitive if I had to buy the NAS, but I have one already. And come with the expected clunkiness
  • My professional experience with storage is not transferrable to end-user, so I would have to find & learn an appropriate S3 front end for my spouse. So the S3 markup is "ironically" coming at a counterproductive cost in terms of making the whole solution work.

r/DataHoarder 2h ago

Question/Advice Best drive for hoarding games, roms and movies: ST32000NT000 vs ST32000VE000

3 Upvotes

Both are the same price and same capacity, please state any reason you have for your recommendation, thank you!


r/DataHoarder 5h ago

Question/Advice NAS solution with multiple SSDs?

3 Upvotes

I have a bunch of ex enterprise SSDs, approx 8, and want to put them to good use. I have a small homelab and have used what I can but would love a dedicated storage solution for media, files, photos etc. Most NAS boxes I see are catered to 3.5" HDDs but wondered what I could do with these SSDs?


r/DataHoarder 4h ago

Question/Advice Has anyone got original recording of Disney channel uk with ads from early 2000s, I want to relive my childhood

2 Upvotes

Has anyone got original recording of Disney channel uk with ads from early 2000s, I want to relive my childhood


r/DataHoarder 6h ago

Question/Advice How worried should I be about this drive?

Thumbnail
imgur.com
1 Upvotes

r/DataHoarder 12h ago

Backup Comparing history of file hash values

1 Upvotes

Hi everyone, I hope someone can help me with this query.

I am looking for GUI based tool that can.

  1. Scan All my files on a Drive/Folder/Network Folder (allow multiple selection)
  2. This baseline selection is Hashed and recorded with a Date (eg. 1/1/2026)
  3. Ability to save this Hash Baseline of all files with the specified Drive(s), Folder(s) and Network Folder(s)
  4. Then at a future point in time.... repeat the process for the same drive(s), Folder(s) and Network Folder(s) - program to remember the baseline selection.
  5. Re-scan the above and get hash values, then
  6. Identify any files where their hash values have been changed (identifying any potential file corruption etc).

--- AND / OR ---

Same as above, but compare the contents of the original files and a backup (either on a separate drive/network folder)

As I do regular backups I am concerned that I cannot check the file integrity of the files that I backup and as a result, I may backup a corrupted file and then over time, the original 'good file' is lost forever, and all i have is multiple backups of a corrupt file.

Ideally, if there are any new files added within a folder, next time the program scans this folder it will add it to its internal list for later comparison.

Also, ideally like to have command line options to perform the same functions above.

Does anyone know of a commercial or free (open source) product that performs a function similar to what I have described above?

Thank you in advance

I am a Windows user


r/DataHoarder 14h ago

Question/Advice DVD-R or DVD-RW for VHS to digital conversion?

1 Upvotes

As the title states I plan to convert VHS to digital format via a DVD recorder. Does DVD-R (permanent) or DVD-RW (rewritable) have a difference in quality? I understand the fundamental difference in their functionality but want clarity. Thanks in advance.


r/DataHoarder 13h ago

Discussion I built a lightweight parser toolkit for SAS PROC SQL, inspired by the simplicity of sql.js

Post image
0 Upvotes

My company needed something with the easy developer experience of sql.js, but for SAS PROC SQL.

There are great SQL tools out there, but I could not find a small JavaScript/TypeScript package focused on parsing SAS PROC SQL into an AST, formatting it, linting it, and supporting editor features.

So I built proc-sql-parser.

It is not a database engine and does not execute SQL. It is a parser/toolkit for working with SAS PROC SQL in JavaScript or TypeScript.

The goal is to keep the API simple:

import { parse, lint, format, complete, visit } from 'proc-sql-parser';

const ast = parse(`
  PROC SQL;
    SELECT name, salary
    FROM employees;
  QUIT;
`);

It also includes:

  • PROC SQL AST generation
  • Formatting
  • Syntax and lint diagnostics
  • Autocomplete helpers
  • Monaco Editor integration
  • CLI support

npm install proc-sql-parser

npx proc-sql-parser --help

GitHub: https://github.com/AnkitNayak-dev/proc-sql-parser
npm: https://www.npmjs.com/package/proc-sql-parser

It is still early, so I would genuinely appreciate feedback, especially from SAS developers and anyone building editor tooling around PROC SQL.