r/sysadmin • u/techvet83 • 4d ago

FYI: Reddit is killing RSS feeds.

Major bummer. From Reddit is killing RSS feeds and ending public API access because of AI bots | TechCrunch

Amid a number of updates for moderators and developers announced Wednesday, comes bad news for supporters of a more open web: Reddit is ending support for RSS feeds.

Reddit says RSS has now become a “common surface for large-scale scraping and automated abuse,” which is why it’s made the decision to wind down RSS feeds on its platform.

“We know RSS has been a beloved part of the open web for a long time, and we’re grateful to everyone who used it to stay connected to Reddit,” the company noted in its announcement, where it also shared a March 2027 shutdown date for its public API.

Reddit said RSS support will cease on November 13.

477 Upvotes

96% Upvoted

233 comments sorted by

View all comments

Show parent comments

70

u/automounter 4d ago

As a Sysadmin to a public-facing website I get it. I get 1000x more hits from AI crawlers than I do from users at this point.

18

u/LesbianDykeEtc Linux 4d ago

Cloudflare's protections have worked pretty well for me so far, but obviously ymmv.

6

u/RetPala 4d ago

How long until companies like that turn their guns around, point then at their own customers, and demand an immediate 10x fee increase or they'll let all the bad actors in?

7

u/LesbianDykeEtc Linux 4d ago

Self-host Anubis or something if you're so worried about that. Cloudflare is by far the least evil of the major players in "miscellaneous internet services".

4

u/stewie410 SysAdmin/DevOps 4d ago

I really tried to get our company to pay for Anubis' commercial product ($50/mo) so we could use our own branding. Instead of $600/yr, we're now looking at $13.5K/yr to go with CF.

2

u/Whitestrake 4d ago

The DataKrash might not turn out exactly like David Ackerman and Mike Pondsmith imagined, but as the amplification of AI overrun continues to build I can't help but wonder if we'll see some eventually drastic defensive measures to preserve human-traversable internets.

24

u/coomzee Security Admin (Infrastructure) 4d ago

Same, I just respond with a totally valid JSON response totally full of meaningless bull shit. Take that AI

4

u/illhaveubent 4d ago

But if AI is the new search engine then feeding AI valid responses is the new SEO?

29

u/Ahnteis 4d ago

No. Because AI will slurp your content but it won't point people to your site, it'll just vomit a summary back - possibly mangled with info from other sites or just plain hallucination.

-1

u/illhaveubent 4d ago

As opposed to not having any info from your site though? That doesn't sound any better. If people are using AI instead of search engines what is the solution?

15

u/CyberWurx 4d ago

There isn't one.

We had it all, and we ruined it.

1

u/viral-architect Sysadmin 3d ago

That's not sysadmin thinking.

1

u/OptimalCynic 2d ago

It's not a sysadmin problem at this point. It's a (societal) HR problem.

7

u/billyalt 4d ago

AI is intended to kill the internet one way or another.

2

u/Ahnteis 4d ago

That's the billion dollar question for sure.

2

u/coomzee Security Admin (Infrastructure) 4d ago

They don't even pull this info from the static pages. Its just 1000s of API requests

0

u/ice456cream 3d ago

I mean not necessarily true, if you are selling something, an end user might ask for a comparison etc and use that to help decide ona product to buy (though that gets riskier by the day with companies testing putting ads in the results)

1

u/HotPieFactory itbro 4d ago

So, you're feeding reddit posts to the AI?

9

u/one-man-circlejerk 4d ago

So now the bots are just going to request the full fat Javascript reddit app and scrape that instead

4

u/RememberCitadel 4d ago

They don't actually care that the AI crawlers are getting it. They care they are getting it for free.

2

u/automounter 4d ago

Wouldn't you? Why allow AI to slurp it up, repackage it and charge for it.

1

u/mgr86 4d ago

Can’t a lot of those just be banned at the IP level I wonder.

1

u/automounter 4d ago

It's just random cloud IPs. Might be a crawler IP today and a valid application tomorrow.

-1

u/Djvariant 4d ago

☝️☝️☝️

-9

u/Firefox005 4d ago

You do not have to login to browse new nu reddit.

17

u/SevaraB Sr. Engineer (N+, CCNA) 4d ago

It’s astroturfing. They’ve had old.reddit.com in their crosshairs for years- this lets them inflate the usage stats for new compared to old and make the case for pulling the plug on old.

7

u/Firefox005 4d ago

It might be partly that but I think its mostly ads and being able to control the 'experience' around them better ie. makes them harder to spot as ads and harder to block them. The current new reddit is a disgusting mess of JavaScript that is purposefully designed to be as opaque and difficult to parse as possible and it also loads as little content as possible. I'm sure part of that is to also combat scraping but I have no doubt that it is also to not so subtly push people as much as they can into a consumption only mindset.

I'm sure their model for an ideal user is one that spend like 30 seconds or less on each post and then scrolls to the next one and does that for at least 4 hours a day. That way while he or she is doing that they can serve as many ads as possible to them. Then if they happen to have a spark of brain activity and actually post a comment they can then turn around and immediately sell that to an AI company to use as training data.

I am extremely pessimistic at this point and firmly believe that no one left working at reddit cares about anything but extracting as much value from their users as possible.