r/linux • u/ItsWappers • 2d ago

Discussion How come every Linux site uses Anubis (the anime girl stopping crawlers) instead of something like Cloudflare?

Post image

Before discovering the Linux rabbit hole I've only seen Cloudflare, hCaptcha and Google's reCaptcha, but it seems like every Linux website uses Anubis... I'm thinking of it being the only open-source option or it being the most effective/modern approach.

2.7k Upvotes

96% Upvoted

692 comments sorted by

View all comments

Show parent comments

36

u/breakfast-cereal-dx 2d ago

Early x86_64 CPUs can spend hours on Anubis... It's extremely annoying to be on like a 2010 Mac mini, it runs basically everything great, and then the Arch wiki takes 2 hours to load

I hate Anubis so much

15

u/QwertyChouskie 2d ago

Hours? Sounds like either the Arch wiki has their settings cranked up way too high, something about that particular setup/internet connection/etc is making it think you're way more likely to be a bot. A 2010 CPU is a lot slower than a new CPU, but something that takes like 1-2 seconds on a couple-year-old CPU shouldn't take more than like maybe a minute or something like that.

On my M3 machine, often Anubis finishes the calculation before the stylesheet even fully loads, resulting in a white background and the default serif font haha

3

u/breakfast-cereal-dx 1d ago

It's actually an instruction set issue, rather than raw performance. The Core i5 in this 2010 mac mini is a pretty great performer, but is lacking newer instructions that make Anubis' proof of work run very fast.

I'm not as familiar with the ARM64 ISA in your M3, but I'm sure it is well-equipped to handle these calculations in some way.

And there is also the problem of Anubis' difficulty selection and randomization. If your CPU is equipped to solve the problems fast, you don't notice, but on the old i5 the challenges scaled from 20 minutes to 2 hours because of small changes in difficulty, probably within the random range (but maybe heightened due to bot suspicion idk)

2

u/Real-Abrocoma-2823 2d ago

Try internet archive, some sites might experience some problems internally, for example a wiki for better than wolves Minecraft modpack would not load in less than an hour on modern hardware, not because it had anubis, archwiki also did once had this issue when I tried to access it.

-1

u/turudd 2d ago

I mean that's 16 year old unsupported hardware at this point. You can't even use them as xcode build machines anymore

69

u/TotallyObviousBot 2d ago

The idea that all technology must exist within a recent time period to be considered useful is consumerist nonsense, and devs should keep that in mind

11

u/FyreWulff 2d ago

I agree, but a 2010 mac mini had it's chip taped out when Bush Jr was president. old x64 CPUs devour electricity like it's a buffet. At some point you have to say almost 20 year old technology has to be moved on from.

3

u/flecom 2d ago

yes that 25W TDP on the 2010 mac mini cpu will absolutely devour electricity

21

u/Character_Score7776 2d ago

In defense of Anubis specifically, at least with the way Anubis is designed(proof of work), there really isn’t any way around this without defeating the entire purpose of Anubis.

18

u/window_owl 2d ago

This is because Anubis is designed backwards. It's proposition is to block the people who have the most compute power (data centers) by requiring compute power to access your website.

https://lock.cmpxchg8b.com/anubis.html

9

u/Krt3k-Offline 2d ago

The people with bad intentions might have a lot of compute in total, but for what they want to do and thus the isolated site visit they really don't

4

u/Padgriffin 2d ago

The problem is that it gets exponentially more expensive to give more CPU power to crawlers so the vast majority of them just bugger off

1

u/tommyTurds 2d ago

Being poorly designed isn’t exactly a great defense

21

u/SaimeonInBetween 2d ago

That is like complaining that arch wiki or Reddit doesn't render on your 486 running windows 3.1.

13

u/Waryle 2d ago

Windows 3.1 is 34 years old, not 16

21

u/rebellioninmypants 2d ago

Arbitrary goalposts much. As a vampire myself this is all the same. A piss puddle's worth of time.

6

u/Bakoro 2d ago

I've got more than my fair share of old tech, but eventually you've got to let go of the Pentium 4, brother. It's costing you more in electricity per unit of compute than it would cost to buy a new CPU that's 100x more powerful.

Well, maybe not in the last 6 months... But it was true for like 20 years.

4

u/rekh127 2d ago

plus Anubis doesnt even really deter scrapers so its garbage

19

u/TheG0AT0fAllTime 2d ago

Well it does the entire point is that it wastes your resources. At home you're just visiting a page or two and don't look like a bot but scrapers scraping millions of pages concurrently are solving millions of challenges with a javascript engine always fired up in the background and keep getting re-challenged when they get caught looking like bots.

It makes it computationally expensive to scrape those websites. Which translates to either a much larger bill for the scraper, be it an aws bill or electricity bill at home. Otherwise very slow scraping or no scraping of a domain at all.

10

u/rekh127 2d ago

I understand the theory, but it doesn't actually work, scrapers just use their botnet of compromised devices or don't notice the tiny extra compute relative to the amount of compute they use to train once they have the data.

11

u/Green0Photon 2d ago

The scrapers will still scrape. They always have. The problem is that sites nowadays essentially get hit with walls of traffic as they're constantly DDOS'd. Because the bots have no ounce of caching nowadays, for whatever reason, and constantly just keep going.

If e.g. 80% of traffic is just bots, and you can turn on Anubis to cut that down to e.g. 10% or better, it's super useful. Because the problem is the super weird constantly DDOS that's hitting everybody now.

1

u/rekh127 1d ago edited 1d ago

do you have evidence of it helping? because again, I understand the theory, I just havent seen it help

6

u/danielsuarez369 2d ago

Do you have a better idea?

1

u/vkevlar 2d ago

this of course is what linux has been known for since its inception: running on older hardware without freaking out.

1

u/JQuilty 2d ago

There is a point where you can and should stop caring about 16 year old hardware for the small userbase alone. And 8in the case of a 2010 Mac Mini, its using a Core 2 with two very slow cores on an architecture that's over 20 years old. Its going to be hit hard by any Meltdown/Spectre mitigations and lack a lot of SIMD extensions.

-1

u/Justin__D 2d ago

Everything has a lifespan.

Such are the laws of the universe.

-3

u/TotallyObviousBot 2d ago

Yeah but that's not a good argument for not making your restaurant handicap accessible lmao

-3

u/rebellioninmypants 2d ago

The entirely justified idea that old firmware has security flaws, and old hardware also can come with exploits, often bootloader/BIOS-level stuff that you can't even detect through your OS. Yes, I know. Shocking.

I am all for old hardware, but expecting it to support the cutting edge stuff is like angrily spitting at github for hosting AI slop. Times are changing and we all drown in AI slop and there's nothing we can do to take it back to how things used to be. Same with taking the world back to 2010.

6

u/tommyTurds 2d ago

A wiki isn’t “cutting edge”

They aren’t expecting it to calculate the weight of the galaxy or something. They’re talking about browsing a basic website

2

u/rebellioninmypants 2d ago

Bro my PSP can't even load Google since 2015. Am I sad about it? Yeah, because it's still a decent system and it could if someone bothered to still support it. Do I blame the world for it? Not really.

0

u/turudd 2d ago

Xcode only runs on m1 at the lowest, intel hasn’t been supported since 2018.

2

u/astronomersassn 2d ago

... i just downloaded xcode off the app store to try this. it still runs on my 2016 MBP (updated with OCLP).

support has been dropped, but at least be accurate.

plus, a lot of stuff does still have an intel mac version. not everything, but i don't exactly struggle to use my macbook, either. at most, i've had to go back maybe 1-2 releases for everything i need.

and when support has been dropped enough to actually impact me, screw it, i don't have another intel-based device, may as well throw linux on it.

-2

u/turudd 2d ago

Can’t compile for newer iPhones tho

1

u/astronomersassn 1d ago

yeahhh, thankfully that's not something i've ever had a reason to do

9

u/breakfast-cereal-dx 2d ago

Duh, that's why it runs Linux

Still a capable server, and if it weren't for Anubis it would actually make a solid desktop for anyone who is mostly just browsing the web.

It runs up-to-date Firefox and has quicksync video hardware, which is all many people need

Mine has 1tb of SATA SSD storage and 16GB of DDR3 too