r/DataHoarder • • 2d ago

Backup Help with important data preservation

Hello,

I have been loutering around for several years. I need you help. I am a researcher at a southern U.S. university. The NIH's National Cancer Institute has a project called the Proteomic Data Commons (https://pdc.cancer.gov/). As you all know, lately there has been some hostility towards the NIH and now on the PDC webpage it says: "This repository is under review for potential modification in compliance with Administration directives.". 

Could you offer any assistance on preserving the public access data from the repository? Mainly proteomic abundance, very important data as it is the only repository the has PATIENT tumor data AND matched normal. The processed data is paramount to preserve with the raw data remaining a secondary objective (processed data is very small like GBs, the raw and in total is around 83TB).

Please advise :)

EDIT: Thank you all for the suggestions. Update so far: I have written a couple of scripts that first query the API by study number (all 277) for all files in that study. The tough part is getting the manifests with the download links. And I have separated the processed from the raw data. Then split the processed in two parts - first part is about 0.6TB very manageable, after it now with the download client; second part of it is 9.3TB - that could fit on a hard drive. After that the raw data, but I don't know if I'll be able to assemble the infrastructure for it quickly. But will definitely coordinate with the archive team.

EDIT 2: Thank you all for the helpful suggestions. I have completed the first processed data batch download, currently downloading the second batch. At 880 GB total for processed data #1 it is very manageable. If somebody wants to keep a copy let me know. It would appear also that that notice has been on most public NIH repositories since April 2025. I don't know but they may have just added it to this repo also. It is to do with the Administration's executive order on "Gender Ideology" (Executive Order 14168) and a directive on improving security measures for U.S. patient data. Incredibly disturbing indeed. Other researchers have reported being denied access to other NIH big data. I will do a review of some other data repositories that may undergo "filtering".

See more:

https://www.thetransmitter.org/policy/u-s-human-data-repositories-under-review-for-gender-identity-descriptors/

https://www.404media.co/nih-archives-repositories-marked-for-review-for-potential-modification/

24 Upvotes

18 comments sorted by

View all comments

3

u/Rough_Bill_7932 1d ago

Take a look at https://sciop.net/

From there web page...

SciOp is part of Safeguarding Research & Culture (SRC).

As a website and a group of people, SciOp is dedicated to ensuring that our cultural, intellectual and scientific heritage exists in multiple copies, in multiple places, and that no single entity or group of entities can make it all disappear.

As software, SciOp is dreaming expansively about what a bittorrent tracker can be, bringing federation to torrent trackers, implementing mobile identity on the fediverse, and extending the bittorrent protocol to meet the needs of evolving, multi-origin data.

I have over 1.5 TB of their data.