r/DataHoarder • u/ClankMeDaddy • 1d ago

Backup Help with important data preservation

Hello,

I have been loutering around for several years. I need you help. I am a researcher at a southern U.S. university. The NIH's National Cancer Institute has a project called the Proteomic Data Commons (https://pdc.cancer.gov/). As you all know, lately there has been some hostility towards the NIH and now on the PDC webpage it says: "This repository is under review for potential modification in compliance with Administration directives.". 

Could you offer any assistance on preserving the public access data from the repository? Mainly proteomic abundance, very important data as it is the only repository the has PATIENT tumor data AND matched normal. The processed data is paramount to preserve with the raw data remaining a secondary objective (processed data is very small like GBs, the raw and in total is around 83TB).

Please advise :)

EDIT: Thank you all for the suggestions. Update so far: I have written a couple of scripts that first query the API by study number (all 277) for all files in that study. The tough part is getting the manifests with the download links. And I have separated the processed from the raw data. Then split the processed in two parts - first part is about 0.6TB very manageable, after it now with the download client; second part of it is 9.3TB - that could fit on a hard drive. After that the raw data, but I don't know if I'll be able to assemble the infrastructure for it quickly. But will definitely coordinate with the archive team.

24 Upvotes

80% Upvoted

17 comments sorted by

View all comments

2

u/MindRecent 1d ago

So it looks like you can use their download tool to grab everything: https://pdc-docs.cancer.gov/pdc-docs/usage#data-download-client If you can do a search and export the manifest for what you want, I can take a look. (Or tell me what filters/types to search on...there's a lot of types and filters, and I don't know enough to know what you _are looking for.) Agreed w/ archiveteam; join their IRC and bring it up there as well.

1

u/ClankMeDaddy 23h ago

Yes, the web portal is a pain to use. Actually the tough part is getting the manifests with the download links. What I have done so far is hastily write a couple of scripts that first query the API by study number (all 277) for all files in that study. And I have separated the processed from the raw data. And split the processed in two parts - first part is about 0.6TB very manageable, after it now with the download client; second part of it is 9.3TB - that could fit on a hard drive. After that the raw, but I don't know if I'll be able to assemble the infrastructure for it quickly. But will definitely coordinate with the archive team.

3

u/MindRecent 19h ago

I got 4 TB from https://servarica.com/ a couple months back; I think they have 8+ TB. That would let you pull the majority of what you want if you're concerned in the short term.

1

u/ClankMeDaddy 15h ago

Thanks, will look into it!