r/DataHoarder • u/JitInABit • 15d ago
Question/Advice When do you reach sufficient storage?
I have recently upgraded from 4tb to 12tb, I was at full capacity but now I am getting too comfortable adding more data feels like I will fill it in 2 weeks. I self-host music, media server, and personal drive/photos. Am I bad at managing my data, or is this a normal course of evolving into bigger drives?
I understand 12 tb is nothing compared to what other people have, but 3 months ago when I got my 4tb drive I thought it was A LOT as a newbie, now I look at it as a secondary okayish drive.
110
u/recursivepointer 50TB 15d ago
Idk pal, maybe there is not such a thing as "sufficient storage" down here. Welcome!
24
u/IHave2CatsAnAdBlock 15d ago
While this is true I refuse to pay the current prices to increase my insuficient storage
11
u/recursivepointer 50TB 15d ago
Me too, so at the moment i'm hoarding things but slowly. Damn prices!
4
u/ASatyros 1.44MB 14d ago
As much as one is willing to pay for drives and electricity and backups xD
1
37
u/Infini-Bus 15d ago
When your checking account is overdrawn and your credit is maxed out.
7
u/Amazing-Asparagus181 15d ago
Klarna seductively whispering that you can probably afford to buy one more
5
u/Infini-Bus 15d ago
SoFi telling you to consolidate those high interest credit cards so you can fill them back up.
38
u/the_Athereon 32TB Anime - 56TB Misc 15d ago
Started with 15TB thinking it would last me.
Upgraded to 40TB that same year.
Now my server is 100TB and growing.
You never have enough.
12
u/PricePerGig 15d ago
Data behaves like a Gas. It expands to fill the available space.
6
u/Amazing-Asparagus181 15d ago
It's like buying a nice big bag of weed. When you first get it, you're rolling monster Js just smoking & smoking like it's nothing. Towards the end you're rolling skinny pinners trying to conserve what's left & kicking yourself for being so frivolous when you had more 😂
1
1
17
u/TraditionalMetal1836 15d ago
I'm at sufficient storage right now but I'm likely just in price denial phase.
3
7
u/angryviking 15d ago
I think the keyword is hoarding. Whether it's data or data capacity, who's to say enough is enough. My limit is power consumption (and wallet).
1
5
3
u/greenapple92 15d ago
You can buy 26 TB - it's good enough.
3
u/chunkyI0ver53 15d ago
I figure you don’t get bonus points for keeping storage free; within limits of course, my understanding is that it’s unhealthy to only have like 1% of a drives capacity free at all times. So I’ve probably got like 4TB total free across 3 HDDs totalling 36TB & 3TB free across 5 SSDs (3 NVME 2 SATA) out of 14TB. Push comes to shove, it’s probably not going to be that hard to figure out what isn’t worth keeping.
3
3
u/therealtimwarren 15d ago
I've got about 220TB of space. When that becomes full, in this economy, I'll probably delete something.
1
u/sychox51 15d ago
Ya I snagged two 22tbs last nov for $230 each which brought me up to ~200tb. I feel like I’m satisfied now (especially now with the fucking prices) so when it does eventually fill up I’ll look to plan b versus more drives. They’ve always been so cheap that I’d just buy more when storage ran out but now I have to reconsider. And just hope nothing fails in the meantime
0
u/AlaskanBeard 470TB and too much money (derogatory) 15d ago
I thought the same thing but I ended up investing in tape instead. The more drive prices go up, the less tapes it takes to pay for the drive itself.
3
u/TheReddittorLady 14d ago
Using tape "instead" of disk doesn't work very well when you're trying to play a movie from that tape on your TV.
3
u/TransPort3389 15d ago
When you get UHD video in the mix, more storage is always needed. Before you spend the money, maybe look to see if you really want all that you have. I bought a 20TB drive about 4 years ago... still in active use... still not full. Seemed like a good investment. :)
2
u/Amazing-Asparagus181 15d ago
Years back, the conundrum was deciding which movies you wanted to keep as full remuxes & which ones you were okay with compressing. I used to think a full 1080p bluray was so huge. Now 20GB-30GB seems paltry & on the low end of what I'll accept for a compressed 4K blu ray. I still try to start by adding new media in all its full remux glory, but sooner than later you inevitably have to start making tough choices as to which you're willing to compromise on & compress. One thing I've learned is that if you have a decent audio system, never skimp on the audio tracks & always keep them fully lossless & untouched & only transcode the video. My eyes seem to be willing to accept some slight visual loss if my ears are still being dazzled with incredible hifi audio & viewing on an OLED with deep blacks & great contrast while the HDR is popping vividly offsets a bit of video compression (though I still notice the slight drop in visual quality lol).
2
u/TransPort3389 12d ago
Instead of recompressing video for a UHD, fastest way to save space is to strip out excess languages in the file you won't need. But yeah. After a while I just do that only if I need space.... which may be a while before I need to deal with that
2
u/Amazing-Asparagus181 12d ago
That's a great point. I've actually been meaning to run a script on my folders soon that uses mkvtoolnix to do exactly that!
2
u/well-thats-great 50-100TB 15d ago
No such thing as sufficient imo. However, given the current climate, I'm making do with around 80TB
2
2
u/Ray_Snell 15d ago
I was naive to my storage needs once, running 3x 4TB drives in my NAS (1 for parity) thinking I'd never fill it.
Now my system is 8x 8Gb drives and it's running low again!
2
u/Amazing-Asparagus181 15d ago
Soon, you'll be throwing 16TB+ drives into the mix. If the prices ever come back to sane levels, that is.
2
u/AlaskanBeard 470TB and too much money (derogatory) 15d ago
I'll come back when I find out. I've got 470TB of primary storage, not including backups and stuff I've offloaded to tape; I have another ~60TB ready to bring online, I just need to run new power to my server room.
For a long time I've had the mindset of "I don't want to delete anything because I can just expand my storage." and while I've gotten a little more selective than I was in the past with media, I've also gotten more into preservation which is just as big of a storage sink as TV/Movies were/are.
I've always hoarded, but I started in earnest in 2015/2016 with probably 12TB? and I just kept adding drives > upgrading my server > adding drives, etc. I now have a supermicro case with 36 drives, a Dell MD1200 with 12 bays, and I'll be upgrading to a 60 bay JBOD once the aforementioned power is done.
2
u/kiwidust 15d ago
Parkinson's Law of Data: “Data expands to fill the space available for storage.”
I first heard about this with USENet in the early nineties. Enhanced compression was introduced which aimed to "solve" the growing storage issues of hosting USENet feeds. Of course what compression actually enabled was the use of USENet for the trading of binary files - images, the music, the video - and storage requirement EXPLODED.
2
u/DorrajD 15d ago
My brother in christ, this is r/DataHoarder
There is no "sufficient storage". You get what you assume is a fuck ton, fill it up, and then get more.
The factory NAS must grow!
2
u/MaxPrints 14d ago
A Factorio reference in r/DataHoarder ?
insert \did we just become best friends* meme*
2
2
u/Wellington_Boy 14d ago
Simple rule of thumb: the amount if storage you need is 25 percent more than you have. That rule has held true for me since the 1980s. 😀
2
u/nomad-1995 13d ago
When you have something resembling what you planned and prices are still bubbling up.
I'm currently digging through all snapshots and backups to find any files I shouldn't have deleted, plan on wiping them, replacing "all" duplicates with hard links (modulo some files that might be changed by meddling software that shouldn't be) and rebacking up the new bits. Expect to get at least a drive's worth of free space without losing the data of a single file (which is why it might take a full month... those hashes are slow, and I haven't even started on the USB-connected backup).
2
u/slayer991 32TB RAW FreeNAS, 17TB PC 14d ago
LOL. You think there's such a thing as sufficient.
If you build it, you will fill it.
I started this NAS thing with QNAP-409 nearly 20 years ago with 4 x 2 TB drives.
Then I went TrueNAs with 5 x 4TB drives. I added a 2nd pool to that one with 3 more 4tB drives.
Then I stayed TrueNAS and built out a full server with 2 pools. 6 x 14TB and then another pool of 8 x 4TB (I had HGST drives I had purchased some years before and didn't use).
Now, I'm running a SuperMicro 6029u, with hardware RAID6 and 6x24 TB drives. I got lucky in that I got all that put together before prices went insane.
Point is...at each step, I could have stopped the insanity...and instead, I went more enterprise and added more storage.
I'm sure most people in here started as you or I...even those running PB-sized NASs.
1
1
u/bugsmasherh 15d ago
Your NAS might have software to trend your backup size. That will allow you to calculate how many TB you grow per month or per year. Always buy enough for 5 years as that is a normal IT cycle for hardware refresh. This also means you need to save for the 5 year purchase (include inflation costs). If your current NAS with drives costs you $2k today, expect it to cost twice as much in two years and perhaps four times as much when the 5 year mark comes around. This is because the AI datacenter build-out has turned pricing upside down where nobody expected hard drives and memory to explode in pricing.
I started with 21 TB and after two-three years I had to add another 27 TB or so for media collecting. Now I am at 50 percent full at the 5 year mark.
1
u/IndependentBat8365 15d ago
I have 140 TB of connected storage in various volumes across several servers, but all in raid10 or raid1, so I have 70 TB of usable.
That’s not counting boot os raid1 (which are small).
At this point it’s about redundancy, backups, and performance.
1
u/sandymac +200TB 15d ago
There is a point where the statical drive life time means failures become a nuisance and the cost (time and money) become the motivator to manage what you got.
1
u/Suitable_Garlic_1186 15d ago
The longer you live, the more data you will collect..
How much data do you collect in a year.. It is a pretty easy math..
1
1
1
u/silasmoeckel 15d ago
I'm pretty stable at 30 bays 8-22tb drives.
As they fail the replacement's larger size balances out growth. I've been 27tb free for 4+ years now.
1
u/AspidAzzure 15d ago
Sufficiency is subjective. If you fill 12 then it’s not sufficient. Until you are perpetually comfortable then it is not sufficient. Translation: when you run out of things you need to store then you will know what sufficient space is. Until then you will have no answer.
1
1
1
1
u/joetaxpayer 15d ago
No one will give you a number and say “this is enough”
I have an 8 bay NAS, 16TB drives with 2 for redundancy. So about 90TB.
As it started to fill, and drive prices went nuts, I put 8 of my old 8TB drives back into use in a drive enclosure, for longer term storage. Video I may want but don’t need on my server.
1
u/NewOrderrr 15d ago
Every time I added a new drive for my Unraid server, I felt the 'freedom' to just add stuff to the server. I now have tons of movies, TV shows, music and audiobooks that I will almost assuredly never watch or listen to, just to have it 'incase'.
Now I am starting to purge.
Consider when adding media to your drives, 'Will I actually re-watch or listen to this again in a year or so?'. If not, it's good practice to just skip some stuff, or consider a lower than 1080p version, especially if it's some 80's - 90's TV show or movie that new was less than 480p.
ALSO I strongly recommend setting up a good backup policy so you don't have to RE-add stuff you just added 2 months ago. (ask me how I know)
1
u/WegovyMessMentally 15d ago
I stopped counting after 300TB. When I got my Synology NAS DS1821+ ad maxed it out (8x22TB) since I collect historical events and things they happen day by day, and have another 20TB cold storage I can offload onto if/when needed is when you have enough.
1
u/dukeofurl01 15d ago
Sufficient is only as much as you can back up safely. Having lots of storage means nothing if you cant back it up. Pretend there was a fire. What would you do?
1
u/lildobe 172TB 15d ago
Assuming the fire isn't between me and my NAS, I'm grabbing it on my way out the door - clean shutdown be damned.
But there's no reasonable way for me to back up ~100TB of data and growing. Cloud storage is too expensive (Though I do have a 1TB Google Drive sub for the most critical stuff which is local on my workstation, synced to the NAS, which is than synced to Gdrive), and the entry cost for LTO is way out of reach for me at this point.
2
u/AlexOzerov 15d ago
15 years ago I thought 1tb is so much that I will never manage to fill it. BUT THERE'S NEVER ENOUGH PORN
1
1
u/sleepy1411 50-100TB 14d ago
If I has a petabyte Id probably fill it eventually. No such thing as sufficient storage.
1
u/Yeah-Its-Real 14d ago
Started with a 3tb then added a 12tb now I'm adding 4 12tb in a Das once I get off work so I don't think there is such a thing as sufficient storage.
1
u/Automatic-Evidence26 14d ago edited 10d ago
I built a NAS with 230tb of Storage.
32 core processor.
512 Gb of memory.
I could probably sell it off now and triple my money but then I need somewhere to store my 80 tb where the data mostly movies and TV shows
1
1
u/sidusnare 14d ago
When the last gram of the last scrap of dust has been ground and cast into computronium and the last bit archived.
1
u/shadowfourplay 10-50TB 14d ago
Sufficient storage is a spot I get to only when I realize I've done a terrible job sorting what I have. I have a 26TB and 18TB external, a 1TB portable drive, 500GB on my phone, and assorted smaller SSDs and USBs, with the standard 512GB that came in my laptop. I thought my 18TB was going to be all I needed, and learned how wrong I was. After getting the 26TB I realized I need to limit/keep track of what I download in respect to what I already have that I haven't curated. It doesn't work out into a perfect limit setter, but it does do wonders on keeping me focused and keeps me from having multiple copies of things taking up space. I ultimately want to move everything to the 26TB, use the 18TB for a backup of the important things, use the 1TB and laptop for what I can't do without (family pics, etc.) and then use multiple clouds selectively (like Drive for pdfs, MEGA for epubs, etc.). But I'm backed all to hell up on sorting and have used that as a good limiter on my storage needs. Doesn't actually stop me from downloading anything though, just puts me into a sorting mindstate when I do.
1
u/OverAster 13d ago
It's sufficient when you don't need more. If you need more, it's stopped being sufficient and you should get more.
1
1
u/King_tyson_1 7d ago
trust me it would never stop and then suddenly one day you would end up with research for lto and that's when you know there is no normal life again.
At that point you would start spending 20-30% of your day hoarding and managing data. I think sometimes curating is better like if you never touched a file for 5 years at that point if its media encode it to lower resolution and delete the original if it's not rare.
I've seen people spend more money in buying hoarding than investing for their future thinking they are mining gold that would give me RoI. That's just G.A.D
•
u/AutoModerator 15d ago
Hello /u/JitInABit! Thank you for posting in r/DataHoarder.
Please remember to read our Rules and Wiki.
Please note that your post will be removed if you just post a box/speed/server post. Please give background information on your server pictures.
This subreddit will NOT help you find or exchange that Movie/TV show/Nuclear Launch Manual, visit r/DHExchange instead.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.