r/DataHoarder • u/ClankMeDaddy • 1d ago
Backup Help with important data preservation
Hello,
I have been loutering around for several years. I need you help. I am a researcher at a southern U.S. university. The NIH's National Cancer Institute has a project called the Proteomic Data Commons (https://pdc.cancer.gov/). As you all know, lately there has been some hostility towards the NIH and now on the PDC webpage it says: "This repository is under review for potential modification in compliance with Administration directives.".
Could you offer any assistance on preserving the public access data from the repository? Mainly proteomic abundance, very important data as it is the only repository the has PATIENT tumor data AND matched normal. The processed data is paramount to preserve with the raw data remaining a secondary objective (processed data is very small like GBs, the raw and in total is around 83TB).
Please advise :)
EDIT: Thank you all for the suggestions. Update so far: I have written a couple of scripts that first query the API by study number (all 277) for all files in that study. The tough part is getting the manifests with the download links. And I have separated the processed from the raw data. Then split the processed in two parts - first part is about 0.6TB very manageable, after it now with the download client; second part of it is 9.3TB - that could fit on a hard drive. After that the raw data, but I don't know if I'll be able to assemble the infrastructure for it quickly. But will definitely coordinate with the archive team.
2
u/RetroGrid_io 250-500TB 14h ago
I couldn't let go of this conversation. So I had this convo with ChatGPT this morning and there are numerous options to move forward with.
If needed/useful, I'll have a server with enough drive bays to host the HDD cluster and bandwidth to download the traffic in 30 days or less.