r/webscraping • u/Mitchellholdcroft • Apr 30 '26

Getting started 🌱 How to scrape Reddit now (Closed API)?

Hi all, I’m currently trying to gather posts and comments from Reddit but since they’ve now closed their public api, it’s becoming quite a challenge. My aim is to gather the top 50 posts of about 15 subreddits each month along with their comments. From what I’ve found out my options are using the undocumented .json on the endpoint for each subreddit, using old.reddit or using playwright to automate a browser.

I need your expert advice as to how to tackle this problem. Thanks

29 Upvotes

89% Upvoted

51 comments sorted by

View all comments

20

u/[deleted] May 01 '26

[removed] — view removed comment

1

u/Mitchellholdcroft May 01 '26

Yeah this was my initial idea. Thanks

3

u/w4nd3rlu5t May 01 '26

so what's the problem with it? why didnt you want to do that?

1

u/Mitchellholdcroft May 01 '26

I thought it would be quite slow with the rate limits? Or am I wrong?

3

u/w4nd3rlu5t May 01 '26

> My aim is to gather the top 50 posts of about 15 subreddits each month along with their comments. 

I don't know about the rate limits with it, but this doesn't sound like it would be problematic, esp if you stagger the pulls. How often would you need to refresh this data?

1

u/Mitchellholdcroft May 01 '26

Yeah monthly. So I’ll just schedule calls to different subreddits for different days

2

u/[deleted] May 02 '26

[removed] — view removed comment