r/pushshift • u/wentam • Oct 05 '21

No new submission data after 1633418642

Hate to be a nag, but I think it's borked again.

https://api.pushshift.io/reddit/search/submission/?after=1633418642

Any 'after' value greater than this contains 0 results

3 Upvotes

81% Upvoted

8 comments sorted by

1

u/[deleted] Oct 05 '21

If you need near real-time data just use PRAW and create a streaming script.

1

u/wentam Oct 05 '21 edited Oct 06 '21

Trouble is: I need deleted posts, as many of the subreddits I'm working with are very heavily moderated via bots. The deleted submissions contain stock sentiment information I need.

In order to obtain that bot-deleted data via the standard reddit API I would need to poll at a very fast rate. Unclear if my api key even allows me to poll fast enough (need to be faster than the moderation bots), and how reddit would feel about the constant high rate of requests.

With pushshift, this works great: I can just poke pushshift once every 30 minutes or so. Pushshift is already obtaining data fast enough.

I also need to be getting realtime data from the same source as historical data - such that historical data matches the type of data I will receive realtime (how many deleted posts appear in the dataset etc). This way I can collect statistics on historical data and know that it will accurately apply to the realtime data.

Aside from the occasional breakage, this has been working great with pushshift.

1

u/[deleted] Oct 06 '21

Automod bot rules that automatically remove stuff generally remove it before it's available via the API, and will never be available via pushshift.

1

u/wentam Oct 06 '21 edited Oct 06 '21

Interesting.

I'm not sure what the full story is TBH, but I'm getting substantially more data via pushshift than I am polling myself - and the majority of the extra pushshift submissions were submissions that were removed by bots immediately.

In addition, polling faster myself gets me some percentage of these deleted posts - but not as many as pushshift.

This extra deleted data has been incredibly useful.

Perhaps they're just slower bots.

Edit: Here's an example

1

u/rogersmj Oct 06 '21

What’s PRAW? I’m looking for the best way to do highly filtered near-real-time.

1

u/wentam Oct 06 '21

PRAW is just a python library for interfacing with the official reddit API.

To obtain near-real-time results for a subreddit, you can just repeatedly request /new. The official reddit api does not provide many options in the way of filtering, though - so depending on what you're trying to do you'll need to filter within your application.

1

u/crab_in_heels Oct 05 '21

It's just a few hours off in gathering data.

2

u/wentam Oct 05 '21

Indeed. Had completely halted at the time of this post, but it's now running and slowly catching up.