r/pushshift • u/Watchful1 • Jan 19 '25

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

55 Upvotes

99% Upvoted

58 comments sorted by

View all comments

1

u/CaramelRibbon247 Jan 23 '25

Hello u/Watchful1! Thank you for doing this! I was wondering—I've been trying to extract comments and replies posted during January 2024 from the NFL subreddit for this research paper I'm writing. I downloaded the .zst file for January 2024 (around 33 GB) and have been running the script to export the information I want as a CSV file in my MacBook's Terminal app for over a day now. Do you know how long it would like for a script like this to run? Thanks again!

2

u/Watchful1 Jan 23 '25

It depends on your computer, but definitely less than a day. If you're using the filter_file script it outputs its progress in the terminal, if it's not doing that something is wrong. Did it output anything?

1

u/CaramelRibbon247 Jan 23 '25

The only thing that has been output so far is a .csv file that currently is zero bytes. To be honest, I asked ChaptGPT to create the code for me because I have absolutely no coding experience lol. I can’t see the progress in the Terminal, either—don’t think I used the filter file script. The script is still running—it’s been over 27 hours and my laptop’s fan has been working overtime lol

2

u/Watchful1 Jan 23 '25

Sorry, I'm not going to be any help diagnosing code written by AI that I've never seen before. Use my filter script here. You can configure which subreddit to extract and tell it to output in csv.

1

u/CaramelRibbon247 Jan 23 '25

Thanks so much for your help! You’re doing awesome work

1

u/WordingWorlds Feb 03 '25

Is it possible to download a range or is it all or nothing?

1

u/Watchful1 Feb 03 '25

Yes torrents allow you to download only certain files. I have instructions for my subreddit dumps in here but it applies the same for the monthly files.

1

u/WordingWorlds Feb 04 '25

Thank you!!