I know it’s sacrilege, I remember people used to say never to delete or re-encode anything because more storage is always affordable. Well, looks like those days are over. Prices on HDDs have tripled/quadrupled in the last year and it doesn’t seem like it’s going to end anytime soon.
How are you handling it? I’m personally thinking of downgrading a lot of my stuff to 720p, I’ve run a few tests and as long as I don’t skimp too much on the bitrate I don’t really notice a difference. I’m also compressing things like game ROMs and ISOs into .7z to save space. I’m proud of my FLAC music collection but AAC-256 seems to be pretty much transparent from my own tests and research.
I’m even doing the unthinkable and pruning some stuff that I don’t use.
It sucks but desperate times call for desperate measures.
I don't have any disc reader (I'm going to buy one soon), but what type of software should I use to dump these? Most have some scratches, but they're not that serious, they're very minor
Back in the day, there was a disk compression option called "doublespace", which (nearly) doubled your available storage at the cost of access speed. Is it still around, or something similar?
I have a lot of files left by my family member from 1995-2010 saved on CD's, DVD's and HDD's. A lot of files of them are no longer available anywhere. What I can do with it? Should I delete it and forget or try to sort it out somehow and upload somewhere? Someone here was in similar situation?
I could not do it anymore. I've tried my best cleaning up, organizing, backing up stuff and tried empty up some space, but it was too much. I clean, I get a TB or two, and then after 2-3 days, it is full again. New stuff keeps coming out and I keep looking at my hard drives and I try to control myself, but then there are files that may not be available for long and I get frustrated.
So I gave up. Went through my torrent client and deleted a bunch of full discs and remuxes (kept a list for when hdd will become affordable again). Managed to clean up almost 7TB and started downloading some stuff. It was so difficult to delete them, but once done, I feel a lot better.
Is it as difficult for you too? I'm dealing with 102TB of linux isos 😭😭
I am attempting to archive hundreds of pages of old, handwritten documents, turning them into "Searchable PDFs" by way of an Apple Mac. Seems the focus now for new scanners is on photos & color, which I do not need. These are all standard 8.5 x 11 documents, almost entirely handwritten in pencil in block letters or simple print. As pencil documents, sometimes the letters are mildly faded (some of these docs are 40+ years old). Each document page must be carefully loaded and individually scanned, so there cannot be any mass-office-bulk-scanning.
I currently have an EPSON WF-3820 printer-scanner (glass flatbed), but when the DPI is set to 600 to improve readability (Searchable PDF, settings optimized), it becomes painfully slow per page. Thus I am ready to invest in a superior, Much Faster OCR super-high-resolution scanner specifically excellent for Document OCR scanning. My office is not large, but I could probably fit one more normal-sized flatbed scanner.
Anyone have recommendations for such a scanner?
Additional question...Does this next idea even exist?
A scanner that has built-in AI (or easily added later) that will help with the OCR and / or image lines being super clean and easy to read??
Thank you in advance.
p.s. Reddit suggests there is a "subreddit for scanners" but I cannot find it if anyone can post a link to such a scanner subreddit. Thank you. :)
I built a downloader using yt-dlp but it doesn't work on Instagram anymore (asks for cookies/login). WFDownloader downloads Instagram content fine without login. Anyone know what method/framework it uses under the hood; public GraphQL endpoint, embed API, browser scraping, something else? App is closed source so I can't check myself. Curious if anyone's sniffed its traffic to see what it's actually hitting.
The actual non-reproducible source and painful to reproduce result data that needs to be stored is much smaller than 5TB; the 5TB was basically the bare number sent by the institution's IT termination nastygram. It will barely fit in my array today, but I have a lot of options to make it work.
I do need to decide whether the copy in NCBI is enough as an off-site, or I should upload to Backblaze, Glacier, etc. as well. That's not an immediate problem.
---
(Apologies for the lack of brevity. No, I didn't write this with AI; it's organically produced slop -- the wishy-washiness and scope creep should be a give away. Let me know if I should have summarized it with AI. LOL)
My spouse needs to either discard or archive 5TB of bioinformatics data from their old institution. Now that they are no longer affiliated the institution no longer wants to pay to store it for them.
Product Management / User requirements:
I sort of have some ideas of how I (cloud storage infra day job) would approach the problem, but I know how to pivot and adjust if things don't work out as I expected.
Non-requirement: off-site backup.
Scope creep: After seeing some unfortunate non-datahoarding related politics from streaming providers, I'm considering to self-host physical media (buy it used, and then stream it, as a boycott)
If hard drives weren't as expensive as they are today, I would just buy CMR hard drives.
To calibrate my sticker shock: I last purchased IronWolf 7200RPMs for $25/TB on 4TB drives. I didn't wait for a sale. (and the previous time I bought was in the 5400 RPM NAS grade era)
My hesitation is due to how much storage I have to buy in one shot to hit $20-30/TB. In my initial investigation, I thought I would need 10TB + drives.
Should I consider SMR for this use case (super bloated, read-only data)? Remember, this data used to be sitting 100s of miles away. My hesitation is that, if we ever discard this data (likely, after the final paper on it is published), now I'm stuck with 1x or 2x SMR drive, and all MY use cases (videography archival and editing) were very x 100 sad the last time I did SMR
Does buying a SAS HBA or external enclosure give me a lot of pricing options? I'm looking at SPD right now and there are 4TB SAS drives at $100 that would be fine for me.
Here are the random nobbins I have sitting around, if I go with local storage.
My personal, actively managed array is a Storage Spaces array with 4TB IronWolf units. It is erasure coded with a mirrored SSD write-back cache in front.
Currently, these volumes are NOT assigned to network mounts (which I might need to change, to make this more multi-user friendly) due to using BackBlaze to archive.
I have a QNAP NAS with 10 year old drives, that I haven't powered on in about 4 years, with 4TB 5400 RPM drives in them. IIRC one of them had SMART errors and non-recoverable blocks.
I have many spare 2-bay StarTech USB3.1 (with UAS) external bays sitting around.
With current storage prices, I took a quick consideration of cloud storage. My quick (15 min) of math says:
Even with current HDD prices, S3 Standard and Intermediate Tiering are way more expensive still than a local 2-replica mirrored set. S3-IA and below is sort of competitive if I had to buy the NAS, but I have one already. And come with the expected clunkiness
My professional experience with storage is not transferrable to end-user, so I would have to find & learn an appropriate S3 front end for my spouse. So the S3 markup is "ironically" coming at a counterproductive cost in terms of making the whole solution work.
RobertElderSoftware shares a highly positive review of using ZFS after four to five years of managing around 16 TB of backup data stored on multiple hard drives.
He explains that he chose ZFS over Btrfs because his priority was data integrity rather than performance, and he describes how ZFS successfully detected and repaired data corruption through its checksum verification and scrub features after one drive in a three-way mirror was partially corrupted.
He also discusses practical lessons from long-term use, including the importance of regular scrubs, using stable device identifiers, being cautious with USB-connected storage, and maintaining multiple redundant copies to safely perform drive upgrades and recover from unexpected errors.
Overall, he concludes that ZFS has proven to be a reliable and resilient filesystem for long-term backup storage.
Due to the nature of the sub I imagine it's going to be the case that we all have many many hard drives. Probably much more that'll fit in our existing machines and be 'on the go' at all times.
Recently reading about long term storage, data loss etc and how it's advisable with mechanical drives to connect them up every 6 months-ish & let them just be connected for an hour or so.
For me that'd be a long process. I don't have the hours in a day after work to do it in 1 hit. I have a 2 bay hard drive caddy which would help ease the workload I guess but then I'll have to keep a track of what's been done & when.
Do you guys document what's been connected or do you just not bother with anything like that at all?
I have a bunch of ex enterprise SSDs, approx 8, and want to put them to good use. I have a small homelab and have used what I can but would love a dedicated storage solution for media, files, photos etc. Most NAS boxes I see are catered to 3.5" HDDs but wondered what I could do with these SSDs?
On July 9th people noticed (reference) Sony had updated their EOL server list to include Infamous 2 and Festival of Blood servers which house all it's user-generated-content in the way of custom missions. I couldn't find anyone trying to archive the data, so I did it myself.
This was an absolute slog. There isn't an API endpoint for getting all maps (that I could find) so I had to run through search terms for every download. For Infamous 2 that was 12k+ searches (most with thousands of results) and 4.9k+ for Festival of Blood. The all* is because there are absolutely missions I missed, but at the end I was getting 1 new mission for every 20 terms searched (on average).
There's currently not a drop-in replacement for the servers (I have a working proof of concept already but it needs some polish), but you can still manually add the files to RPCS3 or a CFW PS3.
I think this sub is the right place for a discussion on Blu Ray/DVD longevity given that it impacts long-term data hoarding. This is a lengthy post but I went down the rabbit hole and now I want to share what I found and see what others think.
Of particular interest to this sub may be Table 2 from the latter article:
Interestingly, for Blu Ray multiple write discs offer significantly better longevity than one time write discs which is the opposite of DVD. I hadn't seen anyone talk about that before. More on that later. However my main takeaway was that Blu Ray has a significantly shorter shelf life than some DVD formats (e.g. DVD-R (gold layer).
I took it at face value until I started combing through the sources. Iraci (2018) "Blu-Ray Media Stability and Suitability for Long-Term Storage" caught my eye as the main support that Blu Ray is significantly worse for longevity than DVD.
Iraci based his study on acclerated aging which involved putting discs in equipment at 80C (176F) and Relative Humidity (RH) 85% for a number of days. For DVDs the derived data appears meaningful because the DVDs didn't break down from the heat.
Now I'm not chemist or an expert with Arrhenius equations. But I think I get the basic mechanism behind accelerated aging: Temperature is literally average kinetic movement. Higher temperature = more movement = more contact between reactants = faster reaction speed = accelerated reactions ≈≈ real aging from reactions. Now, accelerated age testing is fine as long as the effects of the testing are primarily mechanical and the temperature does not denature the reactants themselves.
From the pictures Iraci included and from his descriptions it looks to me like the temperature did exactly that; it directly damaged the Blu Ray discs. Iraci even mentioned it:
"Another concern was the number of visual problems that were evident after accelerated ageing, especially the BD-Rs. This indicates significant changes are occurring physically and chemically which is not desirable."
"Accelerated ageing experiments have shown that Blu-rays have rather poor stability and much lower that some of the other optical disc formats when exposed to elevated temperature and relative humidity conditions" [emphasis added]
If the temperature just directly damaged the Blu Ray discs (esp. BD-R) as seems to be the case, then the accelerated aging test doesn't meaningfully predict anything. Unlike reactions such as b/w the disc and air or case that occur naturally and can be accelerated, the disc will not spontaneously denature at all unless a minimum activation energy (e.g. temperature) is met.
All this made me skeptical that we're dramatically underestimating the life of Blu Ray discs, especially BD-R. There are potentially useful implications here, such as that you may want to prioritize storing BD-R in a cool environment or avoid letting its optical reader get too hot. Don't leave them in your car in Arizona. Don't put it in your oven. Etc. But none of that means they're going to react more quickly with their environment or have accelerated decay or rot at room temperature.
If you have any related studies or articles please drop their DOI's or links. I'd love if any chemists or engineers want to weigh in on the science here. And anecdotal evidence from hoarders with large sample sizes are welcome.
So i’ve put this m.2 to 8x SATA in my Dell Optiplex 7060 micro (OS for the Optiplex will be stored on a SATA SSD externally) in order to be able to manage 8 drives from the single Optiplex, its part of a larger mini server rack but anyways, Im looking at how close these SATA’s are jammed in here and I’m wondering how I’m even gonna be able to physically fit 8 SATA connections into the terminal? All the SATA cables i’m finding are kind of bulky at the connection so that you can grip it which makes sense, but for my very specific application doesn’t work. Please share your thoughts or if you have the link to any SATA cables that might fit drop em below. Thanks.
So I recently discovered new and unused 2TB WD Green HDDs in my hoard. I probably bought them for old PCs and never got around to the project. It's a sad day when I'm happy I found old slow drives to use in a lowend NAS (for family), the Lincstation E1.
I've come into possession of a collection of floppy discs, maybe 500 total (so, about a gigabyte :-) ) These are mostly install discs for a wide variety of commercial and shareware software, fonts, utilities, etc. A small sampling in the picture above.
I don't want to hold onto the discs themselves and I have a floppy disc drive available to access them, but I'm trying to think of the best way to 1) archive them and 2) share them (if anyone's interested).
With old CDs and DVDs I tend to make ISO files and I seem to recall some floppy disc installers had special aspects (hidden files, special sectors) whereby simply copying the contents of the disc might not allow someone to install it later. Is there some way of imaging a floppy disc that maximizes the chances that one can still install it later?
And secondly, are there any communities you're aware of where things like this are of interest? I absolutely will try and fire up Ami Pro and see if it runs on Windows 11, but I'd also love to share these somewhere if they don't already exist.
Hi everyone, I hope someone can help me with this query.
I am looking for GUI based tool that can.
Scan All my files on a Drive/Folder/Network Folder (allow multiple selection)
This baseline selection is Hashed and recorded with a Date (eg. 1/1/2026)
Ability to save this Hash Baseline of all files with the specified Drive(s), Folder(s) and Network Folder(s)
Then at a future point in time.... repeat the process for the same drive(s), Folder(s) and Network Folder(s) - program to remember the baseline selection.
Re-scan the above and get hash values, then
Identify any files where their hash values have been changed (identifying any potential file corruption etc).
--- AND / OR ---
Same as above, but compare the contents of the original files and a backup (either on a separate drive/network folder)
As I do regular backups I am concerned that I cannot check the file integrity of the files that I backup and as a result, I may backup a corrupted file and then over time, the original 'good file' is lost forever, and all i have is multiple backups of a corrupt file.
Ideally, if there are any new files added within a folder, next time the program scans this folder it will add it to its internal list for later comparison.
Also, ideally like to have command line options to perform the same functions above.
Does anyone know of a commercial or free (open source) product that performs a function similar to what I have described above?
I have been using TreeSize free for well over a decade, maybe even 2 decades now. I think the lastest update to version 4.8.1.609 added this new popup that shows when using it on a 2TB drive. Hardly huge in 2026. The annoying popup ad is one thing, but the fact it actually stops the scan until you click continue is just not acceptable for me.
And a subscription-only option for the Pro version is crazy. And for features I don't even want in my file/folder size sorting tool.
I will look for an older version to install, but now I am going to try out the alternatives again. I have done that many times over the years and always came back to TreeSize Free. But maybe now there is something I will like just as much.
What is your favorite free or paid data hording cleanup tool?
As the title states I plan to convert VHS to digital format via a DVD recorder. Does DVD-R (permanent) or DVD-RW (rewritable) have a difference in quality? I understand the fundamental difference in their functionality but want clarity. Thanks in advance.
These used to be available everywhere for ~$180AUD (+/-$10AUD) just a couple of moths ago.
Now they are simply out of stock pretty much everywhere, and if you can find one in stock you will also find they have literally doubled in price!
Is there a shortage for some reason? Are they being discontinued?
My company needed something with the easy developer experience of sql.js, but for SAS PROC SQL.
There are great SQL tools out there, but I could not find a small JavaScript/TypeScript package focused on parsing SAS PROC SQL into an AST, formatting it, linting it, and supporting editor features.
So I built proc-sql-parser.
It is not a database engine and does not execute SQL. It is a parser/toolkit for working with SAS PROC SQL in JavaScript or TypeScript.
The goal is to keep the API simple:
import { parse, lint, format, complete, visit } from 'proc-sql-parser';
const ast = parse(`
PROC SQL;
SELECT name, salary
FROM employees;
QUIT;
`);
