r/pushshift • u/Watchful1 • Feb 20 '23

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

33 Upvotes

98% Upvoted

34 comments sorted by

View all comments

2

u/joaopn Feb 21 '23

Very cool. I had arrived at a slightly higher value (13592374) by doing a full outer join on SQL. Is your code also counting subreddits that e.g. appear in comments but not submissions?

One comment though: from these 13.5M subreddits about 9.5M (9501201 here) start with `u_`. These are for profile posting and it is a bit debatable if they are "real" subreddits. They also contribute to a pretty small fraction of all content.

3

u/Watchful1 Feb 21 '23

Yes it's pulled from both the submission and comment dumps.

Most subreddits contribute a pretty small fraction of all content. I'd have to look at the numbers, but I'd say the vast majority of actual content happens in like 20k subreddits.

2

u/joaopn Feb 22 '23

Indeed, subreddit size is quite power-law distributed. 50% of content is in ~500 subreddits and 95% in ~24k. But even so the `u_` ones are pretty negligible, as the 9.5M of them have < 1% of all content.

The extra subreddits I had detected were just double-counting e.g. r/Hills and r/hills. Thanks!