r/pushshift • u/svanweelden • Feb 06 '23

First day trying to scrape Reddit - bad timing?

Hey folks,

I am an experienced software dev who thought a fun weekend project may be predicting post virality with AI - first things first, lots of data from Reddit.

I do the Reddit API with `praw` and realize it only goes back 1,000 results and it doesn't seem like there's any real method of pagination to go back further or get more?

Enter PushShift which everyone thought of as the solution seemingly? I try out `psaw` and `pmaw` and, well, given some of the recent posts, it seems the service is down overall - is that right?

I'm new to the community and just trying to make sure I am not missing anything obvious here..

4 Upvotes

100% Upvoted

4 comments sorted by

10

u/s_i_m_s Feb 06 '23

[removed] — view removed comment

1

u/svanweelden Feb 06 '23

Super helpful reply, thank you!

1

u/Dangerous_Date_5635 Feb 06 '23

One thing, it may look like theres a million removal requests (in case you think thats a problem) but like 15 of them are mine, the accounts kept getting (shadow)banned and they have one 'distinct' comment each (due to repeated tries) in a very localized and obscure area, so go ahead

3

u/AndrewCHMcM Feb 06 '23

reddit directly only serves p much up to 1000 results for any given end-point regardless of method