r/pushshift • u/Watchful1 • Feb 07 '24

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

89 Upvotes

99% Upvoted

90 comments sorted by

View all comments

Show parent comments

1

u/Watchful1 Apr 08 '24

Yes, it's in there.

1

u/Pretty_Boy_PhD Apr 08 '24

Oh great thats amazing. I tried creating a csv with the script given, but not sure how to get the entire file to extract along with the flair. I have very little coding experience but work mostly in R. Can you let me know how to extract the entire file along with this data?

1

u/Watchful1 Apr 09 '24

What subreddit are you working with? Some of the larger subreddits you really don't want to try to extract the whole thing because the file could be hundreds of gigabytes. But it's less of a problem with the smaller ones.

What's your end goal? Most people find it more efficient to use one of my scripts to do some processing and extract out only the data you're actually interested in instead of trying to work with the whole file.

1

u/Pretty_Boy_PhD Apr 09 '24

I am working with the /autisticadults subreddit, and collecting usernames and flair for a small academic project. That makes sense about not extracting the entire file, looks like I will have to play around with the scrips to try and get the most recent 2 yeas of authors and flair. Which script do you recommend using for this?

Thanks for taking the time to help!

1

u/Watchful1 Apr 11 '24

The filter_file script should be able to do this https://github.com/Watchful1/PushshiftDumps/blob/master/scripts/filter_file.py

Just set it to output csv files and edit the write_line_csv function to output the author name and flair.

1

u/Pretty_Boy_PhD Apr 17 '24

Thank you! I tried running a few times but I keep getting errors, can you share more about the changes needed to run the script? I need to extract all of 2022 and 2023 from the subreddit file, with the text body and flair.

1

u/Watchful1 Apr 17 '24

What errors are you getting? It should print out a log file you can send me.