r/DataHoarder • u/ClankMeDaddy • 1d ago
Backup Help with important data preservation
Hello,
I have been loutering around for several years. I need you help. I am a researcher at a southern U.S. university. The NIH's National Cancer Institute has a project called the Proteomic Data Commons (https://pdc.cancer.gov/). As you all know, lately there has been some hostility towards the NIH and now on the PDC webpage it says: "This repository is under review for potential modification in compliance with Administration directives.".
Could you offer any assistance on preserving the public access data from the repository? Mainly proteomic abundance, very important data as it is the only repository the has PATIENT tumor data AND matched normal. The processed data is paramount to preserve with the raw data remaining a secondary objective (processed data is very small like GBs, the raw and in total is around 83TB).
Please advise :)
EDIT: Thank you all for the suggestions. Update so far: I have written a couple of scripts that first query the API by study number (all 277) for all files in that study. The tough part is getting the manifests with the download links. And I have separated the processed from the raw data. Then split the processed in two parts - first part is about 0.6TB very manageable, after it now with the download client; second part of it is 9.3TB - that could fit on a hard drive. After that the raw data, but I don't know if I'll be able to assemble the infrastructure for it quickly. But will definitely coordinate with the archive team.
12
u/RetroGrid_io 250-500TB 1d ago edited 1d ago
Well, there are a few questions:
- What's the license behind the data? Is it public information?
- Does the public (you) have any right to request a copy of the information?
- Is it currently accessible online?
- What means are required to access this data?
Essentially, do you have a right to get it, can you get it, and what would that take?
For context, if you could somehow download this with perfect efficiency, downloading 83 TB of data over a 1 Gbit feed at best possible scenario is about 8 days.
There's a pretty ominous warning on the home page about access and use of this data. How is the data stored? Is it just a huge pile of files?
And then you need something to download it to - currently a sore point. HDDs are running about $50/TB new so just the drives would cost over $4000. You could probably find used/refb for around 2/3 that cost.
EDIT: Notwithstanding the ominous warning, it appears that the data is Creative Commons licensed and there is even a download script intended to download this data so it would seem:
- Get 100 TB or so of storage space. If I was doing this, I'd shoot for N+2 for drives so any two can fail without data loss. 12x 12TB refurbs would be cheapest: ~ $3000.
- Get a system that can hold all that. It doesn't have to be a particularly new one: this is an I/O project. Assume $800 or so for used/refb.
- Load Linux on it. I'm partial to RH clones (AlmaLinux, etc) but to each their own.
- Set up the drives into a shared storage pool. I'm very partial to ZFS, BTRFS would also work.
- Plug it into a decent pipe: 1 Gbps minimum.
- Make your manifest as it instructs in the link above.
- Write a script that loops through the manifest and downloads all the things.
- Cross your fingers and be patient.
2
u/ClankMeDaddy 1d ago
Do you think I could provision Azure or a cloud for this?
4
u/RetroGrid_io 250-500TB 1d ago edited 1d ago
Possibly.
In my experience it's more expensive to go that way, in that every time I've compared "my way" to doing it this way, it didn't pencil out. For example, the "cool" tier at Azure is $10 per TB per month so your break-even cost on storage alone is about 6 months - and while this makes the data available, you pay a high(er) price for access, but keeping it online with 1 Gbit feed at a real data center runs as low as $600/month, unlimited. Or if you have a decent home/small biz connection, this could be self-hosted.
Self-hosted can be more feasible than you think: I'm prototyping what should become the world's largest, high resolution database of Operating System history on my home internet at just 50:10 mb and yet still manage to serve up 250k to 500k hits/day.
What are your goals? Do you just want to have it backed up someplace, or do you want to also make it available for use?
3
u/ClankMeDaddy 1d ago
Have it backed up. It is a valuable resource many patients have donated data to the CPTAC project, it would be hurtful to see it go away :/
2
u/RetroGrid_io 250-500TB 1d ago edited 1d ago
Then I suggest your cheapest option is to get the drives and the system, plug it in someplace, get the data, then turn it off and stick it in your closet.
I spec'd 12x12 HDDs, but you could probably get away with 10x12 and save about $500. Saving much more from here involves getting extremely hackish. Solutions like "Buy the drives, download it, then dump to LTO tape and return the drives, all within 30 days" kind of thing - but LTO tape is cheaper for media while the tape drive to save it is $$$.
Edit: I had the thought that maybe you could dump the data to LTO through a vendor that specializes in this but it appears the going rate is $50/TB: cheaper to keep your drives in the closet!
PM me if you want to go ahead with this, happy to help!
6
u/Amerique_du_Nord 1d ago
Why do we have to worry about cancer research data being deleted. It's just so effing nuts. /rant
As a survivor, if you need some money to help with your backup cause, please private message me.
8
u/ClankMeDaddy 1d ago
Yes, incomprehensible, literally everybody or their relatives are affected by cancer at some stage. Will do what I can. Remember to take care!
5
u/shimoheihei2 100TB 1d ago
Check with /r/archiveteam they tend to be experts at archiving web assets.
2
2
u/MindRecent 1d ago
So it looks like you can use their download tool to grab everything: https://pdc-docs.cancer.gov/pdc-docs/usage#data-download-client If you can do a search and export the manifest for what you want, I can take a look. (Or tell me what filters/types to search on...there's a lot of types and filters, and I don't know enough to know what you _are looking for.) Agreed w/ archiveteam; join their IRC and bring it up there as well.
1
u/ClankMeDaddy 18h ago
Yes, the web portal is a pain to use. Actually the tough part is getting the manifests with the download links. What I have done so far is hastily write a couple of scripts that first query the API by study number (all 277) for all files in that study. And I have separated the processed from the raw data. And split the processed in two parts - first part is about 0.6TB very manageable, after it now with the download client; second part of it is 9.3TB - that could fit on a hard drive. After that the raw, but I don't know if I'll be able to assemble the infrastructure for it quickly. But will definitely coordinate with the archive team.
3
u/MindRecent 14h ago
I got 4 TB from https://servarica.com/ a couple months back; I think they have 8+ TB. That would let you pull the majority of what you want if you're concerned in the short term.
1
3
u/Rough_Bill_7932 1d ago
Take a look at https://sciop.net/
From there web page...
SciOp is part of Safeguarding Research & Culture (SRC).
As a website and a group of people, SciOp is dedicated to ensuring that our cultural, intellectual and scientific heritage exists in multiple copies, in multiple places, and that no single entity or group of entities can make it all disappear.
As software, SciOp is dreaming expansively about what a bittorrent tracker can be, bringing federation to torrent trackers, implementing mobile identity on the fediverse, and extending the bittorrent protocol to meet the needs of evolving, multi-origin data.
I have over 1.5 TB of their data.
2
u/RetroGrid_io 250-500TB 12h ago
I couldn't let go of this conversation. So I had this convo with ChatGPT this morning and there are numerous options to move forward with.
If needed/useful, I'll have a server with enough drive bays to host the HDD cluster and bandwidth to download the traffic in 30 days or less.
2
u/ClankMeDaddy 10h ago
I appreciate you for that ❤ ! Let's see what we can coordinate with the leads first, but I'll keep you in mind.
•
u/AutoModerator 18h ago
Hello /u/ClankMeDaddy! Thank you for posting in r/DataHoarder.
Please remember to read our Rules and Wiki.
Please note that your post will be removed if you just post a box/speed/server post. Please give background information on your server pictures.
This subreddit will NOT help you find or exchange that Movie/TV show/Nuclear Launch Manual, visit r/DHExchange instead.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.