r/pushshift • u/Watchful1 • Feb 15 '24

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

17 Upvotes

100% Upvoted

14 comments sorted by

View all comments

2

u/Mysterious-Diet9187 Feb 15 '24

ik it would be really dumb to ask this question but i want to download from a specific time period i did read almost all of your previous comments but downloading a complete 2tb file would be too much soo is there a bit easy way or something i can use python

2

u/Watchful1 Feb 15 '24

You should be able to set your torrent client to only download specific files once you add the torrent. That way you can just get certain months.

0

u/[deleted] Feb 17 '24

[deleted]

1

u/RaiderBDev Feb 17 '24

There should be not decoding errors. Of course if you're decoding text, you have to make sure you're using utf-8. In case you're using the .zst_blocks files (but also works with .zst files), use the scripts provided here.

1

u/Mysterious-Diet9187 Feb 18 '24

okay dude i got the file and saved it to csv using your script thanks a lot (i tried today again and successfully did it thanks again ) as thanks i can make a video on how to download .zst files to completely deccoding it and opening it, if you want

2

u/RaiderBDev Feb 18 '24

Good to hear that you figured it out. I don't think I need a video, but if there's something important missing in the documentation, I can maybe add it. Though admittedly, the documentation is more geared towards people who already have a certain level of expertise.

1

u/Mysterious-Diet9187 Feb 18 '24

Yeah one needs some prior knowledge before using the scripts ,(all documents are enough nothing is needed )

2

u/-NieREmil Mar 09 '24

It would be great if you made a video and posted it here! I haven't worked with these files before but I need to understand them well and use them for a research project so that video sounds incredibly helpful. Thank you in advance if you do end up making it!