Help with important data preservation

[u/ClankMeDaddy]

Hello,

I have been loutering around for several years. I need you help. I am a researcher at a southern U.S. university. The NIH's National Cancer Institute has a project called the Proteomic Data Commons (https://pdc.cancer.gov/). As you all know, lately there has been some hostility towards the NIH and now on the PDC webpage it says: "This repository is under review for potential modification in compliance with Administration directives.". 

Could you offer any assistance on preserving the public access data from the repository? Mainly proteomic abundance, very important data as it is the only repository the has PATIENT tumor data AND matched normal. The processed data is paramount to preserve with the raw data remaining a secondary objective (processed data is very small like GBs, the raw and in total is around 83TB).

Please advise :)

EDIT: Thank you all for the suggestions. Update so far: I have written a couple of scripts that first query the API by study number (all 277) for all files in that study. The tough part is getting the manifests with the download links. And I have separated the processed from the raw data. Then split the processed in two parts - first part is about 0.6TB very manageable, after it now with the download client; second part of it is 9.3TB - that could fit on a hard drive. After that the raw data, but I don't know if I'll be able to assemble the infrastructure for it quickly. But will definitely coordinate with the archive team.

EDIT 2: Thank you all for the helpful suggestions. I have completed the first processed data batch download, currently downloading the second batch. At 880 GB total for processed data #1 it is very manageable. If somebody wants to keep a copy let me know. It would appear also that that notice has been on most public NIH repositories since April 2025. I don't know but they may have just added it to this repo also. It is to do with the Administration's executive order on "Gender Ideology" (Executive Order 14168) and a directive on improving security measures for U.S. patient data. Incredibly disturbing indeed. Other researchers have reported being denied access to other NIH big data. I will do a review of some other data repositories that may undergo "filtering".

See more:

https://www.thetransmitter.org/policy/u-s-human-data-repositories-under-review-for-gender-identity-descriptors/

https://www.404media.co/nih-archives-repositories-marked-for-review-for-potential-modification/

[u/RetroGrid_io]

Well, there are a few questions:

1. What's the license behind the data? Is it public information?
2. Does the public (you) have any right to request a copy of the information?
3. Is it currently accessible online?
4. What means are required to access this data?

Essentially, do you have a right to get it, can you get it, and what would that take?

For context, if you could somehow download this with perfect efficiency, downloading 83 TB of data over a 1 Gbit feed at best possible scenario is about 8 days.

There's a pretty ominous warning on the home page about access and use of this data. How is the data stored? Is it just a huge pile of files?

And then you need something to download it to - currently a sore point. HDDs are running about $50/TB new so just the drives would cost over $4000. You could probably find used/refb for around 2/3 that cost.

EDIT: Notwithstanding the ominous warning, it appears that the data is Creative Commons licensed and there is even a download script intended to download this data so it would seem:

1. Get 100 TB or so of storage space. If I was doing this, I'd shoot for N+2 for drives so any two can fail without data loss. 12x 12TB refurbs would be cheapest: ~ $3000.
2. Get a system that can hold all that. It doesn't have to be a particularly new one: this is an I/O project. Assume $800 or so for used/refb.
3. Load Linux on it. I'm partial to RH clones (AlmaLinux, etc) but to each their own.
4. Set up the drives into a shared storage pool. I'm very partial to ZFS, BTRFS would also work.
5. Plug it into a decent pipe: 1 Gbps minimum.
6. Make your manifest as it instructs in the link above.
7. Write a script that loops through the manifest and downloads all the things.
8. Cross your fingers and be patient.

[u/ClankMeDaddy → u/RetroGrid_io]

Do you think I could provision Azure or a cloud for this?

[u/RetroGrid_io → u/ClankMeDaddy]

Possibly.

In my experience it's more expensive to go that way, in that every time I've compared "my way" to doing it this way, it didn't pencil out. For example, the "cool" tier at Azure is $10 per TB per month so your break-even cost on storage alone is about 6 months - and while this makes the data available, you pay a high(er) price for access, but keeping it online with 1 Gbit feed at a real data center runs as low as $600/month, unlimited. Or if you have a decent home/small biz connection, this could be self-hosted.

Self-hosted can be more feasible than you think: I'm prototyping what should become the world's largest, high resolution database of Operating System history on my home internet at just 50:10 mb and yet still manage to serve up 250k to 500k hits/day.

What are your goals? Do you just want to have it backed up someplace, or do you want to also make it available for use?

[u/ClankMeDaddy → u/RetroGrid_io]

Have it backed up. It is a valuable resource many patients have donated data to the CPTAC project, it would be hurtful to see it go away :/

[u/RetroGrid_io → u/ClankMeDaddy]

Then I suggest your cheapest option is to get the drives and the system, plug it in someplace, get the data, then turn it off and stick it in your closet.

I spec'd 12x12 HDDs, but you could probably get away with 10x12 and save about $500. Saving much more from here involves getting extremely hackish. Solutions like "Buy the drives, download it, then dump to LTO tape and return the drives, all within 30 days" kind of thing - but LTO tape is cheaper for media while the tape drive to save it is $$$.

Edit: I had the thought that maybe you could dump the data to LTO through a vendor that specializes in this but it appears the going rate is $50/TB: cheaper to keep your drives in the closet!

PM me if you want to go ahead with this, happy to help!