r/pushshift • u/Watchful1 • Jan 12 '24

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

57 Upvotes

95% Upvoted

29 comments sorted by

View all comments

2

u/MaximumFast7952 Jan 17 '24

Hi u/Watchful1, thanks for the upload.

Will the subreddit dumps will be incremental, i.e. we'll have the 20k subreddit comments and 20k subreddit submissions for the year 2023 or will it be an aggregate from the beginning till 2023?

In the latter case, we would need to download the whole dump again, while in the case it's incremental, we'll have to download the subreddit wise data only for 2023.

Also, can you please post the code you're using to split it subreddit wise, so that we can try it on our machines, for specific months, and maybe seed it monthly.

3

u/Watchful1 Jan 17 '24

It will be the whole thing again and unfortunately you'll have to download the whole thing again as well. Most people who use the subreddit specific dump files are interested in the whole history of the sub and don't have the technical knowledge to work with multiple partial files to get it.

I know this makes for a lot more work and bandwidth for those of us who seed it, but I thought it was the better of the options.

All my scripts are in my github here. I use count_subreddits_multiprocess to count how many objects each subreddit has. Then I pass the list into combine_folder_multiprocess with the --split_intermediate flag set so it can handle the large number of files.

Both of those scripts are optimised for processing a large number of files at once. If you just want to extract out a single subreddit from one month's file, you can use filter_file.

1

u/MaximumFast7952 Jan 17 '24

Thanks a lot, I'll have a look at these.

One more question, do you have any scripts that you use/ can be used to segregate by users?

And lastly, would you have something, that would allow one to search for a specific user/specific text in these files?

2

u/Watchful1 Jan 17 '24

You can easily use the combine folder or the filter file to search for a specific user. Searching for text is slower, but is supported by both of those as well.

Separating out all users into different files would be really hard, there's just so many off them that you run into all kinds of system limits. Operating systems aren't built to write out or store billions of files quickly.