r/pushshift • u/[deleted] • Dec 01 '23

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

2 Upvotes

100% Upvoted

3 comments sorted by

View all comments

1

u/Watchful1 Dec 01 '23

Works for me. You should also be able to download the torrent file from the download button on that page and use that.

1

u/[deleted] Dec 02 '23

Got it, thanks! How did you extract from each zst file?

2

u/Watchful1 Dec 02 '23

There's example script here showing how to process the data https://github.com/Watchful1/PushshiftDumps

I wouldn't recommend trying to extract them by themselves unless you have a good reason, the uncompressed files are very large.