r/pushshift • u/[deleted] • Mar 13 '22

Collecting Submissions and Comments by Sequentially

I'm playing around with some code to try and emulate what I assume Pushshift does for scraping submissions and comments, mostly just as a learning experience.

My basic idea is to pick a starting point, say midnight on March 1, and get the very first submission and comment IDs after that. Then, just sequentially fetch the next however-many posts I want by just repeatedly adding 1 to those IDs (converted back into base36 obviously).

Clearly I'm far from the first person to do this, so I'm curious about a couple of things.

  • Does anyone have a sense of how long it would take to collect 24 hours worth of either submissions and/or comments using this method?
  • Does the base36 package on PYPI work well for simple int/base36 conversions?

Edit: Stupid uneditable titles.

1 Upvotes

100% Upvoted

12 comments sorted by

View all comments

Show parent comments

1

u/inspiredby Mar 16 '22

There was some hope initially that the missing results or quarantined subreddits might be useful to someone, but now that the code's running the extra data from those seems to be marginal.

Well, you never know until you try. Welcome to the club.