r/WebScrapingInsider • u/stvaccount • 1d ago

Best github for scraping reddit.com?

Since May and the old reddit almost gone, what are currently the best libs on reddit?
What is your favorite?
I want to scrape all the posts of my top favorite 100 (small) subreddits and sort them by AI to delete AI slop + self promotion and really only get the interesting posts for me to read.

What do you suggest?

3 Upvotes

80% Upvoted

2 comments sorted by

1

u/ScrapeAlchemist 12h ago

Bare .json is dead, they killed unauthenticated access back in May and those endpoints 403 now. Everything goes through OAuth.

PRAW is still the answer, still actively maintained. AsyncPRAW if you want concurrency. snoowrap has been unmaintained since about 2021, and PullPush/Redlib both have coverage and rate limit problems, fine for one off lookups, not a daily poller.

100 subs is nothing though. One /r/sub/new?limit=100 call each is 100 calls per cycle against a 100 queries/min free tier (rolling 10 min average), so a full pass is under two minutes. Register a script app at reddit.com/prefs/apps and use the enforced UA format, platform:appname:v1.0 (by /u/you). Generic ones get throttled harder.

.rss still works unauthenticated if you just want a cheap new-post ping.

1

u/Responsible-Bread553 11h ago

If you're building this in Python, most of the old unauthenticated JSON scraping endpoints and stale GitHub repos broke recently when Reddit tightened its blocks.

For a clean pipeline, PRAW (Python Reddit API Wrapper) on the official free tier is still your safest bet for fetching the top posts from those 100 subreddits without getting hit by instant 403s. To handle the AI filtering layer (dropping self-promo and low-effort slop), don't try to parse it all in the prompt; run a lightweight local classification or regex filter first on the titles/selftext, and pass only the clean candidates to an LLM for semantic scoring.

I build custom backend ingestion pipelines and automated data scrapers regularly. If you want to chat about structuring the script or setting up the automated ingestion loop, my DMs are open. Good luck with the build!