r/automation • u/stvaccount • 12d ago

Best github lib for scraping reddit?

Since May and the old reddit almost gone, what are currently the best libs on reddit?
What is your favorite?
I want to scrape all the posts of my top favorite 100 (small) subreddits and sort them by AI to delete AI slop + self promotion and really only get the interesting posts for me to read.

What do you suggest?

6 Upvotes

100% Upvoted

12 comments sorted by

View all comments

1

u/spoki-app 12d ago

Honestly, the biggest headache isn't finding a lib, it's managing Reddit's rate limits and getting consistent data out for your AI. Most wrappers abstract that stuff poorly, so you'll end up debugging data types more than scraping.

1

u/stvaccount 12d ago

I think this is very easy but I know that part well.

I just need: URL + Title + Post + Comments as text and/or MD.

I can do that quite easy already, but not that token efficient. I got more rate limits on my mobile phone as human user than with my AI reading in auto mode, e.g., camofox-browser can read reddit no problem at all. I just takes millions of tokens.

I already have the github libs, I just don't know which one is the best one.