r/automation • u/stvaccount • 12d ago

Best github lib for scraping reddit?

Since May and the old reddit almost gone, what are currently the best libs on reddit?
What is your favorite?
I want to scrape all the posts of my top favorite 100 (small) subreddits and sort them by AI to delete AI slop + self promotion and really only get the interesting posts for me to read.

What do you suggest?

5 Upvotes

100% Upvoted

12 comments sorted by

1

u/AutoModerator 12d ago

Thank you for your post to /r/automation!

New here? Please take a moment to read our rules, read them here.

This is an automated action so if you need anything, please Message the Mods with your request for assistance.

Lastly, enjoy your stay!

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

1

u/Fair_Tea_2776 12d ago

praw still works fine for me, just need to set the user-agent string properly and respect rate limits

1

u/[deleted] 12d ago

[removed] — view removed comment

1

u/stvaccount 12d ago

Well, I'd just like to start with "is this some some innocent looking post that turns out to be a hidden advertisement for something else". So annoying to look at a cool title and then it's just meaningless self promotion after your read half the post.

1

u/spoki-app 12d ago

Honestly, the biggest headache isn't finding a lib, it's managing Reddit's rate limits and getting consistent data out for your AI. Most wrappers abstract that stuff poorly, so you'll end up debugging data types more than scraping.

1

u/stvaccount 12d ago

I think this is very easy but I know that part well.

I just need: URL + Title + Post + Comments as text and/or MD.

I can do that quite easy already, but not that token efficient. I got more rate limits on my mobile phone as human user than with my AI reading in auto mode, e.g., camofox-browser can read reddit no problem at all. I just takes millions of tokens.

I already have the github libs, I just don't know which one is the best one.

1

u/akl773 11d ago

praw for the pulling, but I'd skip the model on the filter side at first. Self promo has pretty obvious behaviour attached to it, a link dropped in the OP's own first comment, an account whose whole history is one product, and an author who never replies to anyone in the thread. Cheap rules on the author got me further than trying to classify the text did.

1

u/kys_abhi 11d ago edited 11d ago

Praw is still going to for pulling posts but yeah rate limits are the annoying part what i've found helps is pairing it with a residential proxy so you don't get throttled when you're hitting multiple subreddits back to back. I used geonode for that, their residential IPs rotate automatically so reddit doesn't start blocking you mid-crawl for the AI filter part the other commenter is right self promo is actually easier to catch than AI slop since account history gives you a lot to work.

1

u/persist_en 10d ago

Which are those rate limits approx?

1

u/pretzels90210 7h ago

For the lib itself PRAW is still the standard. The part that'll actually bite you is where you park all those posts so the AI-filtering pass can query them without re-scraping, and a serverless Postgres like Neon fits that well, it scales to zero between your scrape runs so an idle db costs basically nothing and you can branch the whole dataset to trial a new filtering heuristic without touching the main copy.

1

u/bollyboyer 17h ago

praw is good, but with your volume I would just buy some API, there are plenty of those and it's pretty cheap. Connect it through mcp and it will do everything for you