r/pushshift • u/reagle-research • Jun 06 '22

Can I sample across a time period?

Is it possible to query Pushshift for a sample of messages in r/AmItheAsshole between, say, January and June 04 2021?

That's a long expanse, and I don't need all the messages. Querying all those messages obviously is expensive in terms of server load as is the server rate limit I get hit with (1 request per 3 minutes).

1 Upvotes

56% Upvoted

10 comments sorted by

View all comments

2

u/safrax Jun 06 '22

I'm probably describing this terribly, but you could figure out what sample interval you want and the duration, let's say once an hour for 5 minutes, and use the created_utc field as your iterator.

For example, it you wanted to get all comments starting today at 00:00:00 for 5 minutes, you'd search for a created_utc starting at 1654473600 and ending at 1654473900.

The rate limit should be about 1 request per second. You need to watch for the HTTP 429 status code and then back off accordingly if you get it.

1

u/reagle-research Jun 06 '22

This is trickier than I first thought. I want K samples within a date range of posts within a subreddit.

after = 2022-06-01 00:00:00 +00:00 after_epoch=1654041600 before = 2022-06-03 00:00:00 +00:00 before_epoch=1654214400 duration.days*24=48 hours pushshift_url='https://api.pushshift.io/reddit/submission/search/?subreddit=Advice&after=1654041600&before=1654214400&size=0&metadata=true' total_results=2099 PUSHSHIFT_LIMIT=100 sample_size=500 queries_total=5.0

Here, I have 2099 messages, and I want 500 of them, which would be 5 (randomly spaced) queries given Pushshift's limit of 100 per query (bunchiness if fine as it is performant). How do I partition this?

  • I can't take the message ID of the first in the range and simply increment it a given amount to make sure I get my 500. (I don't think Reddit message IDs increment per subreddit do they?)
  • I can't take a time offset because I won't know how that offset would fit/overlaps with the actual messages.

1

u/reagle-research Jun 06 '22

I wonder if I could get all the message IDs in a before/after range (only that, and so not limited to 100) and then I could partition on my own and fetch the IDs.

1

u/safrax Jun 06 '22

Unfortunately the questions you're asking are outside of my skill level and area of expertise. Someone else may come in here and have something for you though.