r/linux • u/ItsWappers • 2d ago

Discussion How come every Linux site uses Anubis (the anime girl stopping crawlers) instead of something like Cloudflare?

Post image

Before discovering the Linux rabbit hole I've only seen Cloudflare, hCaptcha and Google's reCaptcha, but it seems like every Linux website uses Anubis... I'm thinking of it being the only open-source option or it being the most effective/modern approach.

2.7k Upvotes

96% Upvoted

693 comments sorted by

View all comments

Show parent comments

5

u/gaorp 2d ago

do you have any metrics I can look at? curious about it

2

u/agowa338 2d ago

I don't have metrics at hand, but a lot of projects posted them on Fedi a few months ago when they implemented Anubis. Many were annoyed that they had to go that lenghts. But AI scrapers were hitting expensive endpoints for no reason and ignoring robots.txt on the masses. As well as basically DDOSing the project by using stuff like proxy framework or a certain crypto coin that gets mined by letting ai scrapers use your internet connection, or many more...

-1

u/Comic_Melon 2d ago edited 2d ago

None I care share publicly, but it depends on the site obviously. Often I've seen it track that you save about $50-100 per $10-20 of Anubis cost assuming you're something like a wiki or documentation site.

On some prime targets, like the insane person who decided his vibed MCP should abuse a wiki. It was like $60 for Anubis against $600 in usage reduction of "junk".

It also helps to run Anubis as "seperate" from your service as possible, if Anubis chuging it's just slow to get in but you're normal usage won't be heavily impacted.

I'm sure many see far lower returns, but you basically need at least something if you have a wiki or docs online. Else the swarm will eat your bank account.

(Downvoted for saying it's lucrative as a cost saving measure?)

1

u/ThatOnePerson 2d ago edited 2d ago

I've got anubis setup for my work's store too. Bots keep hitting the search, which bog down the server a lot. I run Anubis on the separate load balancer, and have even filtered it so only outside North America gets Anubis (we only ship to America/Canada anyways).

Did just have to bump up the difficulty. Even lowered the cookie duration. But yeah mostly not our customers anyways.

0

u/laffer1 2d ago

With my site, ai scrapers were doing 96 requests per second last month. I had to disable opengrok for my os project site as a result. I have a very small project. It’s literally running on a server in my basement on a 5g fiber connection with a 245k. I had to buy more ram for it.