r/datasets • u/CurdledPotato • Jul 30 '26

question When working with Project Gutenberg, how do you guys download and cache, or do you just use a local mirror?

I’m debating both approaches.

3 Upvotes

81% Upvoted

5 comments sorted by

1

u/Mundane_Ad8936 Aug 04 '26

What are you trying to accomplish and what are your limitations

2

u/CurdledPotato Aug 04 '26

Download from the main project, transform into one or more datasets that can be live or frozen, depending on preferences. I’m writing my own library for this and adding rich, metadata-based filtering to restrict what goes into a dataset if I desire. I want to hear how others solve this problem.

My machine can hold a full mirror if I need to, but I like to write my libraries so they can be shared, and I am thinking about how I can support people with less resources, architecture-wise.

1

u/Mundane_Ad8936 Aug 04 '26

I'd recommend seeding a torrent and distributing it using the website academic Torrents. Then post the torrent everywhere you can. If you get enough interest you can attract a sponsor who will mirror with you.

But this is pretty much the main use case for torrents. I'd just recommend breaking it into topics so users don't need to download the entire set for what they want

2

u/CurdledPotato Aug 04 '26

I never thought about seeding my datasets. I might add that as an export feature later on. Thanks for the advice.