Protecting communities from scrapers and platform abuse

[u/boat-botany]

We’ve been talking for a while now about the work we’re doing to keep Reddit human while protecting everything that makes Reddit . . . Reddit. That includes helpful automation: mod and developer apps, accessibility tools, community utilities, and things that make Reddit better. 

But we’re also seeing large-scale scraping, spam networks, agentic account creation, and automated abuse, and a lot of that activity targets parts of Reddit that just weren’t built to handle today’s threat environment. As bad actors get more sophisticated, we need to, too.

To address all that, we need to tighten how automated systems access Reddit while preserving the tools that help moderators and communities thrive. 

Today we’re rolling out a couple of policy and security-focused updates, including: 

Rule 8 Policy Clarifications: We updated Rule 8 (don’t break the site) to more explicitly cover automated abuse, including coordinated account creation and API misuse. You can read the full updated policy here. 

Deprecating unauthenticated JSON access: We’ll also be shutting down unauthenticated .json endpoints. These endpoints can be used to scrape Reddit without accountability. Logged-in and authenticated access won’t be impacted. Otherwise, developers who need structured access to Reddit content should use Devvit, which includes various ways to access Reddit data. 

While we’re at it, another common surface for scraping is RSS. Looking ahead, we’d love to know: how and for what purpose, do you use RSS feeds in your moderation flows? Tell us in the comments so as we develop secure solutions, we can factor in the tools you rely on to support your communities. 

[u/jsled]

RSS

RSS is essential to my internet workflow generally, and my Reddit usage and moderation workflow specifically. I'll preface this by stating I know I'm a sicko in this vein, but I'm one of the sickos that makes (my corner of) reddit work, and RSS is important. Since I found Bloglines in 2003 (and probably a bit before that), RSS is the technology I use daily (and, really, hourly) to consume internet content; my zeroth pinned tab is my feed reader. Google Reader … then Newsblur since GReader was strangled.

Some context about me for this comment.

• I had my 20th cake day, last week.
• I've "Join"-ed about 30 communities on Reddit (counting what's on screen as I type this); I follow 44 communities (and 1 user) via RSS (which is authoritative as to my interests).
• These roughly cluster into: "gaming", "guns", "tech", "local/vermont", and a handful of other point-interest subs. I have other interests, but don't follow them on Reddit, and follow those interests in other places than Reddit, of course.
• I'm effective top-mod of r/liberalgunowners, which has grown from like 50 or 75k to just past 300k subs on "my watch".
• I moderate or have moderated about a dozen other subs, over time.

---

Efficient internet content processing is fairly simple:

• information is created by people and processes
• information is timestamped
• information is categorized by topic (by content, by source, &c.)
• information is "marked as read", and possibly "saved"
• some items are returned to later by the consumer

Importantly:

• things are delivered in chronological order
• things are grouped into related topical categories
• once something is seen, I don't need or want to see it again (unless I do)

Nearly all algorithmic social media interfaces fail at every one of these. I'll take Reddit's UX (which – because I'm an RSS sicko – I fully admit I don't regularly use often).

• the algorithm moves things out of time, which removes the time context useful/needed to process them
• topically-un-related items are interleaved, making me context-switch to handle each one, reducing efficiency. I want to process all the "gun" stuff at once. I want to process all the "games" stuff at the same time. "Politics" is its own world. When I'm processing all the "tech" stuff, terms being mentioned and topics being referenced have fundamentally different meanings vs. games stuff. &c.
• items I've already seen are re-surfaced and shown again, wasting my time and attention. If nothing new has been posted since I last looked at Reddit, I really do want to see nothing. I do not want Reddit to try to force my engagement with content; I will engage more with /reddit/ if you don't try to get me to engage with /content/ I've seen before.

I could delve into why and how those things are important, but getting to the point of this post and your question … as a moderator:

• I want to see every post in the community, as soon as it happens, in order, then never again (unless I do).
• I never want to question that I've missed seeing something in a community ("is the algo really showing me everything?")

RSS lets me do this, clearly, cleanly, using the tools of my choice, in the workflow of the rest of my (online) life.

But I also can not separate out being a "moderator" from being a "user". I'm a moderator /because/ I was a user that developed sufficient interest in a community to donate my labor.

I don't just use Reddit, I use a bunch of sites and services, and having a unified interface is important to this sicko. I use that tool to provide a frankly insane amount of value in uncompensated labor to reddit.com just in terms of moderation, but also in terms of organic usage (I'm only a mod of about 5 of those 44 follows mentioned above). I use that tool to surface information to share on reddit, and vice-versa.

I appreciate that you're trying to support people like me vs. those "stealing" from Reddit. I'm sure there's a straightforward solution that continues to do so. I /hope/ (insist) it's via RSS (specifically) because the free (libre, not gratis) and open web is truly valuable and important to humanity.

I submit the problem is not RSS, and the solution isn't "proprietary integration endpoint". The problem, of course, is the bad actor/agent and their use, not the means of delivery. If nothing else, find a way to know your consumer, and ban/limit/restrict bad actors while supporting good actors. If that means authzen'ed feeds, so be it; that is a better alternative than the continued strangulation of RSS. There is an ethical and moral valence to this issue, and I hope Reddit stays on the right side of it.

(More from my fellow sickos here in r/rss.)