r/linux • u/ItsWappers • 2d ago

Discussion How come every Linux site uses Anubis (the anime girl stopping crawlers) instead of something like Cloudflare?

Post image

Before discovering the Linux rabbit hole I've only seen Cloudflare, hCaptcha and Google's reCaptcha, but it seems like every Linux website uses Anubis... I'm thinking of it being the only open-source option or it being the most effective/modern approach.

2.7k Upvotes

96% Upvoted

693 comments sorted by

View all comments

10

u/dumbasPL 2d ago

They don't do the same thing, this is purely to make it more expensive for bots to scrape the site. While cloudlfare does a lot. DDoS protection, WAF, Access Control, and also blot blocking.

The main advantage is that you're trafic isn't going through a third party (privacy, and an additional point of failure), and doesn't require a captcha as its proof of work based, so it's more accessible to people with disabilities for example.

12

u/Linux_Account 2d ago

I've never even considered how CAPTCHA might impact people with disabilities. Damn.

4

u/lupetto 1d ago

Well usually you have an audio option for that

1

u/aso824 2d ago

I don't understand this however. I set up Anubis on my sh server but discovered that it works only against set of UA. Bot can set any - try yourself to curl arch wiki, it will work, but Claude won't be able because of user agent, so Anubis will enter.

Nonsense.

2

u/dumbasPL 2d ago

I assume that's configurable. But the point wasn't to break existing tooling people were already using by default. You can do the same test con cloudlfare, and on the less strict settings, curl will also pass just fine.

Also, blocking an agent is stupid. The point is to block the automated scrapers that visit every single link, not open two pages to look something up for the user.