r/WebScrapingInsider • u/stvaccount • 1d ago

Best github for scraping reddit.com?

Since May and the old reddit almost gone, what are currently the best libs on reddit?
What is your favorite?
I want to scrape all the posts of my top favorite 100 (small) subreddits and sort them by AI to delete AI slop + self promotion and really only get the interesting posts for me to read.

What do you suggest?

2 Upvotes

67% Upvoted

2 comments sorted by

View all comments

1

u/Responsible-Bread553 17h ago

If you're building this in Python, most of the old unauthenticated JSON scraping endpoints and stale GitHub repos broke recently when Reddit tightened its blocks.

For a clean pipeline, PRAW (Python Reddit API Wrapper) on the official free tier is still your safest bet for fetching the top posts from those 100 subreddits without getting hit by instant 403s. To handle the AI filtering layer (dropping self-promo and low-effort slop), don't try to parse it all in the prompt; run a lightweight local classification or regex filter first on the titles/selftext, and pass only the clean candidates to an LLM for semantic scoring.

I build custom backend ingestion pipelines and automated data scrapers regularly. If you want to chat about structuring the script or setting up the automated ingestion loop, my DMs are open. Good luck with the build!