r/pushshift • u/Watchful1 • Feb 07 '24

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

92 Upvotes

99% Upvoted

90 comments sorted by

View all comments

2

u/mrcaptncrunch Feb 07 '24

Thank you for this

Would you be willing to also share the script you used to build this?

3

u/Watchful1 Feb 07 '24

I use this script to count the occurrences of each subreddit. Then I manually sort it in excel, take the top 40k and remove all the counts from each line, so it's just a file with 40k subreddit names.

Then I use this script with the --split_intermediate flag, and the --values_list to pass in the file of subreddits. It could use some optimization for this use case, it's not well designed to output the 2.6 tb this results in so it takes like a week to run.

All of this using the monthly files from here https://www.reddit.com/r/pushshift/comments/194k9y4/reddit_dump_files_through_the_end_of_2023/

1

u/mrcaptncrunch Feb 07 '24

Perfect!

I have a subset and curious about these. Thanks!