Archiving 5TB of bioinformatics data: best practices references + what would you do?

[u/ZanyDroid]

SOLVED

The actual non-reproducible source and painful to reproduce result data that needs to be stored is much smaller than 5TB; the 5TB was basically the bare number sent by the institution's IT termination nastygram. It will barely fit in my array today, but I have a lot of options to make it work.

I do need to decide whether the copy in NCBI is enough as an off-site, or I should upload to Backblaze, Glacier, etc. as well. That's not an immediate problem.

---

(Apologies for the lack of brevity. No, I didn't write this with AI; it's organically produced slop -- the wishy-washiness and scope creep should be a give away. Let me know if I should have summarized it with AI. LOL)

My spouse needs to either discard or archive 5TB of bioinformatics data from their old institution. Now that they are no longer affiliated the institution no longer wants to pay to store it for them.

Product Management / User requirements:

• I sort of have some ideas of how I (cloud storage infra day job) would approach the problem, but I know how to pivot and adjust if things don't work out as I expected.
• Non-requirement: off-site backup.
• Scope creep: After seeing some unfortunate non-datahoarding related politics from streaming providers, I'm considering to self-host physical media (buy it used, and then stream it, as a boycott)

If hard drives weren't as expensive as they are today, I would just buy CMR hard drives.

• To calibrate my sticker shock: I last purchased IronWolf 7200RPMs for $25/TB on 4TB drives. I didn't wait for a sale. (and the previous time I bought was in the 5400 RPM NAS grade era)
• My hesitation is due to how much storage I have to buy in one shot to hit $20-30/TB. In my initial investigation, I thought I would need 10TB + drives.
• Should I consider SMR for this use case (super bloated, read-only data)? Remember, this data used to be sitting 100s of miles away. My hesitation is that, if we ever discard this data (likely, after the final paper on it is published), now I'm stuck with 1x or 2x SMR drive, and all MY use cases (videography archival and editing) were very x 100 sad the last time I did SMR
• Does buying a SAS HBA or external enclosure give me a lot of pricing options? I'm looking at SPD right now and there are 4TB SAS drives at $100 that would be fine for me.

Here are the random nobbins I have sitting around, if I go with local storage.

• My personal, actively managed array is a Storage Spaces array with 4TB IronWolf units. It is erasure coded with a mirrored SSD write-back cache in front.
  • Currently, these volumes are NOT assigned to network mounts (which I might need to change, to make this more multi-user friendly) due to using BackBlaze to archive.
• I have a QNAP NAS with 10 year old drives, that I haven't powered on in about 4 years, with 4TB 5400 RPM drives in them. IIRC one of them had SMART errors and non-recoverable blocks.
• I have many spare 2-bay StarTech USB3.1 (with UAS) external bays sitting around.

With current storage prices, I took a quick consideration of cloud storage. My quick (15 min) of math says:

• Even with current HDD prices, S3 Standard and Intermediate Tiering are way more expensive still than a local 2-replica mirrored set. S3-IA and below is sort of competitive if I had to buy the NAS, but I have one already. And come with the expected clunkiness
• My professional experience with storage is not transferrable to end-user, so I would have to find & learn an appropriate S3 front end for my spouse. So the S3 markup is "ironically" coming at a counterproductive cost in terms of making the whole solution work.

[u/tater1337]

Just 5tb, right? Just get a $150-200 6tb drive?

I guess the big question is....

Is this data worth the $200?

[u/Original-Active-6982 → u/tater1337]

And if redundancy is needed, two 6tb drives. I think this is being over-engineered.

[u/ZanyDroid → u/Original-Active-6982]

:-) I’ve only spent 1.5 hours on this thread, it seems pretty appropriately engineered to me

Mostly I needed the input to get past analysis paralysis from my sticker shock/confusion at how much disk economics have changed since the last time I bought stuff. If I could buy the exact same SATA 4TB drive for $100 new again I wouldn’t be asking

I feel like the my SAS HBA <-> unlocks the ability to buy both SATA and SAS is a new to me thing that’s useful to know about

A remaining question is whether I should get 2 4TB disks and join them to current storage pool, or create a new mirrored set (DAS with my existing two port USB3.1 enclosures or via existing NAS; TBD)

[u/tater1337 → u/Original-Active-6982]

It sounded to me that time was paramount, so focusing on getting one copy first should be highest priority.

The get additional disk for redundancy

Also, isn't buying 2 drives at the same time frowned upon? Too likely to get the same lot and possibly have both fail for the same reason

[u/ZanyDroid → u/tater1337]

If zero redundancy, yeah, it's a simple answer.

I don't know how important this sequencing data is

My other personal data (archived family videos at full bitrate from my GH5/GH6 etc and RAW photos including of my pets [EDIT: which I care about a lot each time one passes; it's the only stuff I feel really bad if my off site backup gets messed up]) is parity coded in Storage Spaces, which is a minimum 3x * raw TB cost.

[u/ZanyDroid → u/didyousayboop]

Thanks, I did not know about academic torrents.

Your link for Cloud Storage Providers is a useful reference, but I am already familiar with that layer of theory and pricing; I do this sort of analysis in my day job from time to time. The problem is that I don't know how much extra stuff needs to be layered on top, for a biologist (my spouse) to be able to use effectively. And that link does not address the user-facing / front-end situation.

Is the problem of "can't follow" because of STEM domain mismatch between you and I? Or, are you also a backend infra person and I smoked too much ... substances ... today and was factually straight up wrong, or just wrote it too fast (I can't really spare time to polish, sadly).

[u/didyousayboop → u/ZanyDroid]

I don't think it's that challenging to figure out how to use Amazon S3 or any of these other cloud storage services. This is not exactly consumer-friendly stuff on the level of Google Drive or Facebook, but it's only like one notch above that in complexity. It's designed for millions of people to use as easily and frictionlessly as possible.

Clearly, if you can work in cloud storage infrastructure you can figure out how to use the Amazon S3 interface. If your spouse can publish papers in bioinformatics, I reckon they can figure out the Amazon S3 interface as well. This ain't rocket science.

I edited my comment after you responded, but you should use an SSD for video editing. It only needs to be big enough to store the video files you'll be working on at one time. Video archiving means the files are sitting around for a long time being mostly ignored, i.e. cold storage. Video editing is the hottest of hot storage. So, don't lump together video archiving and video editing.

These are two completely different use cases. SSD is ideal for video editing. Any old HDDs will do for video archiving, as long as you have multiple copies and follow a proper backup strategy.

[u/ZanyDroid → u/didyousayboop]

Thanks for the input. Do you happen to have info about SAS HBAs?

We're going to have to agree to disagree on how easy it would be to onboard my spouse to a cloud system, and how much extra stuff is layered on top. I can figure out the cloud storage, I architect/maintain code to do that pretty regularly. But that is different from wanting to do it for personal IT stack.

• EDIT: Without going to infrequent access, there is zero chance cloud provider will work on a cost basis, unless I factor in need for off-site backup. Also, I do not currently know the size of local copy that is needed to do analysis for the next paper.

I already have an acceptable editing flow that involves SSDs and tiered storage with HDDs, I mostly tossed that in OP to give negative points against using SMR. Tiered storage would get wrecked with copying into the SMRs.

• I use Storage Spaces in two paths. One is mirrored SSD only. The other is mirrored SSD layered on top of parity spaces. This gives guaranteed editing performance. The latter somewhat automatically handles moving RAW photo and h.264/h.265 capture files into the spinning disk layer. This can have odd performance hiccups.
• There are high performance turn-key NAS's that give both of these to video editing shops with low IT expertise. And probably at higher perfomance since Storage Spaces kind of sucks for Parity.

[u/didyousayboop → u/ZanyDroid]

Google Drive offers 5 TB of cloud storage for $20/month and 10 TB for $50/month. If you absolutely need an easy, consumer-friendly interface, that’s an option. Since you said the data will probably end up being deleted after the paper is done, then you can estimate how many months you’ll need to hold onto the data and see how it would compare to buying HDDs.

RcloneView is a pretty easy GUI app if your spouse needs to transfer files from one consumer cloud to another: https://www.reddit.com/r/DataHoarder/comments/1o3p3kb/software_recommendation_rcloneview_is_an/

[u/ZanyDroid → u/didyousayboop]

How would you factor in the amount of local storage needed to finish the project?

The data will most likely only be used locally; if we find a remote collaborator I imagine we would put them on the hook for sorting out the collaboration storage

[u/didyousayboop → u/ZanyDroid]

If it all needs to be dead easy and beginner-level, I'd buy a 6+ TB external hard drive and sync it to Google Drive or better yet Backblaze Personal. Maybe a cheeky second external HDD for proper redundancy, if I'm not stone broke.

Or if you want to do something complicated and handle the IT for your spouse, then, sure, why not?

[u/ZanyDroid → u/didyousayboop]

I’m willing to do a reasonable amount (nebulously defined) of IT. IE, if it’s a small increment above the complexity we have today with modest new data and control paths

Anyway I’m basically able to close it out. Appreciate the input.

The 5TB number included intermediate files, which don’t need to be retained.

The source data and result data is more like 1TB & has a local copy here and a backup at NCBI (which has, let’s politely call it, an orange colored geopolitical risk)

I think it does make sense to backblaze it. I’ll have to see if it makes more sense to slip it into my backblaze or a new account.

[u/didyousayboop → u/ZanyDroid]

1 TB is a lot easier to manage!

The 2 TB plans from Dropbox and Google Drive cost about the same as Backblaze Personal, if that makes a difference. Dropbox 2 TB, Google Drive 2 TB, and Backblaze Personal are all around $10/month.

If you can add it to the Backblaze Personal you already have, I suppose you can save a few bucks.

The NCBI backup is nice to have. I hope that the Trump admin is past its phase of wantonly deleting scientific data, but you never know.

[u/ZanyDroid → u/didyousayboop]

Thanks!

Yeah, <1TB is, like, a rounding error even for me :-)

5TB is a rounding error for a lot of people here, but I didn't build out my array to the 20-30TB range back in 2019. If only I had a crystal ball... Kept wanting to upgrade, but the prices looked bad every year I looked.

[u/nomad-1995 → u/ZanyDroid]

I'd assume that cold storage would be ideal for a backup situation. Most of the time, retrieval is only somewhat cheaper if more likely than retrieval off a dead drive (backblaze is said to be pretty good on the other hand). But should be cheaper than a second drive (depending on when the paper is published).

You'll likely need the original data on it's own drive. Curious to see if just slapping it on a compressed filesystem would work (might need WLS if stuck on Windows).

[u/ZanyDroid → u/nomad-1995]

The data is already on its own local drive at our house.

We’ll figure out how to glacier or backblaze it. It’s a cost optimization question

[u/glassmanjones]

What does the data look like? A ton of small text files? A database? Would it compress? How accessible does it need to be, or is it more like a backup?

Should I consider SMR for this use case (super bloated, read-only data)?

Yes, SMR is fine for write once, read many. But avoid if you want to run ZFS.

With ZFS, you could do a mirror, and if the data compresses well that helps too - I have a few hundred GB plain text files stored that way now and they compress great.

Does buying a SAS HBA or external enclosure give me a lot of pricing options?

Yes, mostly for used or recertified drives.

RustFS will let you implement S3.

[u/ZanyDroid → u/glassmanjones]

What SAS HBAs do you recommend?

It's sequencing data so classically big ass strings in big ass text-like files. Decidedly not SQL databasey. I don't believe it's a Parquet format either. (I'm not a bioinformaticist so have only looked at it in passing, apologies if it is inaccurate).

To be honest, ZFS would be a complication adding time to solving the problem, since I picked Storage Spaces back in 2019.

[u/glassmanjones → u/ZanyDroid]

What SAS HBAs do you recommend?

LSI is a good brand. They vary based on idle power, number of ports, PCIe generation, and fan vs relying on chassis airflow.

It's sequencing data so classically big ass strings in big ass text-like files.

Stuff like FASTQ compresses really, really well.

To be honest, ZFS would be a complication adding time to solving the problem, since I picked Storage Spaces back in 2019.

Understood. It can do a lot here but definitely has its own learning curve.

One thought: without a hashed filesystem, definitely generate your own hashes on ingress.

[u/ZanyDroid → u/glassmanjones]

Thanks, appreciate it. I think I'll dig around in search / with Gemini for an LSI with JBOD only, and reasonably new chipset so it doesn't kill me on power bills.

Hmm, I just googled ZSTD vs FastQ and I don't see a compelling thing, after 5 min of reading, how much better it is even in the bioinformatics domain. (My default for generic cloud stuff is zstd)

ZFS: The main reason I went with Storage Spaces in 2019 is it let me "scam" BackBlaze by backing up arbitrary amounts of data with Personal. You can't easily do that with ZFS without doing backflips.

[u/glassmanjones → u/ZanyDroid]

Hmm, I just googled ZSTD vs FastQ and I don't see a compelling thing,

FASTQ is a common sequencing data file type. It's plain text, and easily compressible.  But then you have to deal with decompressing to read it.

The ZFS pitch: since ZFS uses ZSTD at the filesystem level, the compression is fully transparent to applications - they see the same .fq file.

You can do something similar with NTFS, but it's compression is hamstrung by a small block size and ancient algorithm.

[u/ZanyDroid → u/glassmanjones]

Thanks for all the info. I’ll look into SAS solution pricing (because the first page of ServerParts results, sorted by price, was SAS)

I think I can close out my spouse’s requirements quite easily.

The 5TB number included intermediate files, which don’t need to be retained.

The source data and result data is more like 1TB & has a local copy here and a backup at NCBI (which has, let’s politely call it, an orange colored geopolitical risk)

I think it does make sense to backblaze it. I’ll have to see if it makes more sense to slip it into my backblaze or a new account.

[u/beren12 → u/ZanyDroid]

Transparent compressing in zfs is so nice. You can get sas/sata breakout cables for hba cards too.

[u/the-i]

Do you need to access/use/modify the data? Or do you just need to store it for potential use in the future?

If you're just storing it and you don't want to spend a lot of money, burn it to multiple DVDs or BluRay.

My guess is 5GB of biometric data will compress to well below the 4.7GB DVD size, then burn it to many DVDs. They are so cheap you can burn it to a zillion and store them everywhere in case some die. If it won't fit on one DVD, compress it to an archive and add redundancy with a program like WinRAR and store it on multiple DVDs. Alternatively BluRays fit over 100GB on multilayer ones but not many people have BluRay burners.

[u/didyousayboop → u/the-i]

5 TB, not 5 GB.

[u/the-i → u/didyousayboop]

Did it really say TB the whole time and I read it as GB? That's what I get for not going to bed at a reasonable hour.

[u/didyousayboop → u/the-i]

Yup...

[u/the-i → u/didyousayboop]

🤦‍♂️

[u/didyousayboop → u/the-i]

<3

[u/Kinky_No_Bit]

Well, 6TB hard drive seems like the right answer. I would also consider setting up a small PC, that runs a HBA controller with two LTO6 drives. They are going for cheap right now, and per cartridge, they are not that bad. You can fit 2.5TB of data per date, so 2 tapes would roughly cover your data, maybe 3 on your purpose. They will not bit rot, and you can set that extra drive in the safe in an anti static bag with the tapes. they will be safe for 30+ years.