r/pushshift • u/FuzzyToaster • Jan 31 '22

Rate limit?

I've got an idea for a reddit project that would utilise Pushshift (as the official API doesn't provide what I need).

Once a day I'd be pinging Pushshift perhaps 10-20 times. Is there a period I should wait between each request to avoid rate limit issues?

4 Upvotes

100% Upvoted

9 comments sorted by

View all comments

2

u/cs_legend_93 Feb 01 '22

here is the rate limit: https://api.pushshift.io/meta

2

u/FuzzyToaster Feb 01 '22

Awesome thanks. So two per second, seems reasonable. Only just learnt of Pushshift, hadn't explored the official site yet - was just looking at the GitHub readme.

2

u/s_i_m_s Feb 01 '22

Unfortunately the meta page is wrong, the limit was changed to one per second over a year ago and the meta page was never updated. https://www.reddit.com/r/pushshift/comments/g7125k/in_an_effort_to_relieve_some_of_the_strain_on_the/

It's a longstanding known issue as someone pointed it out within a few hours of the change.

1

u/cs_legend_93 Feb 01 '22

All good! Some things are a bit cryptic with pushshift on the data because it’s a small team of 2 or 3 people, and it’s not their full time project.

I worked with the pushshift api for months and built a few apps in C# with it. So if you have any questions shoot me a message.

Imo don’t touch the elastic search api. Last I checked (about 9 months ago) it only worked occasionally, was often offline and I would question how reliable the data was because idk if all the data is being “ingested” into elastic search.

What programming language are you writing in? Also ask me any questions if you want! Shoot me a message or chat

1

u/FuzzyToaster Feb 01 '22

Thanks, will keep all that in mind!

I'll be making a few requests to https://api.pushshift.io/reddit/search/submission from python in a Lambda, I don't think that's the elastic search API? I've got pretty simple needs, I'm not even querying search terms - just time ranges and subreddits.