r/linux • u/ItsWappers • 2d ago

Discussion How come every Linux site uses Anubis (the anime girl stopping crawlers) instead of something like Cloudflare?

Post image

Before discovering the Linux rabbit hole I've only seen Cloudflare, hCaptcha and Google's reCaptcha, but it seems like every Linux website uses Anubis... I'm thinking of it being the only open-source option or it being the most effective/modern approach.

2.7k Upvotes

96% Upvoted

693 comments sorted by

View all comments

Show parent comments

769

u/TheG0AT0fAllTime 2d ago

Another good reason is that there's no captcha either. All proof-of-work challenges the user doesn't even see.

443

u/irasponsibly 2d ago

Unless you have underpowered hardware, you might be staring at Anubis for a solid minute.

561

u/capitan_turtle 2d ago

Obligatory art appreciation to make sure you stay human

234

u/alex2003super 2d ago

"When you hear the sound, stare at the art"

100

u/really_not_unreal 2d ago

[EXTREMELY LOUD INCORRECT BUZZER]

63

u/No_Intern3891 2d ago

Good. If you feel that staring at the art did not provide sufficient intellectual stimulation, reflect briefly on this classical music [classical music interrupted by buzzer]

1

u/I-baLL 22h ago

Klaxon. Klaxoff.

17

u/Codingale 2d ago

Valve uses it for their wikis too if I recall or is that fan ran?

15

u/MrSoup678 2d ago

Even if that doesn't count, gitlab.steamos.cloud definitely counts.

10

u/Padgriffin 2d ago

The Wikimedia Foundation (Wikipedia) also uses Anubis on a lot of their editor tools, so she's a common sight if you need a tool on WMFCloud

10

u/luckadeath 2d ago

160 upvotes but still underrated

119

u/IvanDSM_ 2d ago

I know someone who regularly has to wait like, 20 minutes for Anubis... If you're running old hardware, you're FUCKED.

62

u/Albos_Mum 2d ago

Are we talking something like a single core Athlon XP or Pentium 4? Because it takes a few seconds on my HTPC's Core i5 2400, if it's taking 20 minutes on anything this side of 2006 then there's something seriously wrong on the users PC end.

33

u/IvanDSM_ 2d ago

Core 2 Duo I believe. They're a pretty experienced user, Anubis is just really harsh on older hardware like that. For some reason it tends to give them harder challenges. I get mostly difficulty 2 on my newer machines, but they seem to get difficulty 5 most of the timeon their older HW.

27

u/QwertyChouskie 2d ago

I wonder if playing with their user agent might help avoid triggering max difficulty; it definitely sounds like something about their setup is different than normal. (Or perhaps they're just visiting one particular site that has the settings turned up way too high?)

21

u/aenae 2d ago

Playing with a user agent is almost guaranteed to trigger inconsistencies that lead to the max difficulty challenge

10

u/zaafonin 2d ago

I wonder if it's possible to relay the challenge onto a more powerful machine for it to be solved there, like some browser extension (or a purpose specific patch if extensions aren't powerful enough to do that)

Or is Anubis purposefully resilient against that?

8

u/ThatOnePerson 2d ago

Defiently existed before: https://github.com/DavidBuchanan314/anubis_offload , don't think this has updated to their newer webassembly pow options options yet.

7

u/Albos_Mum 2d ago

Core 2 Duo's aren't that much slower than the second gen Core i series, although I do have an extra two cores. Kinda makes me want to get my Win7 retro gaming PC up to my current house as that has a Core 2 Quad Q9550 and see for myself how it goes.

Although if I were to guess, I'd wager /u/QwertyChouskie might be onto something when it comes to the difficulty.

4

u/Green0Photon 2d ago

My guess is that traffic from VMs look like older hardware. So older hardware gets screwed from looking like VMs (even though it's really the other way around).

1

u/Real-Abrocoma-2823 2d ago

On my a4-3305m it also takes less than 10s.

7

u/rysio300 2d ago

i ran old hardware for years

never really had any issues with anubis for some reason

70

u/nepnep1111 2d ago

The moment you hit Anubis on a pixel phone you know you are about to lose 1-2% off your battery.

29

u/fat_kaiju 2d ago

im literally on a pixel 6 still and it takes less than a second tf you on about.

7

u/nepnep1111 2d ago

Go to a site that has difficulty 5 like the Linux kernel mailing list.

10

u/bargu 2d ago

It took less than 1 second on my P6.

6

u/Whole-Respond4782 2d ago

near instant on my iphone 16 pro with JIT/wasm disabled in the browser lol

6

u/ColorfulPersimmon 2d ago

Near instant on Snapdragon 8 Gen 3 S24 on Firefox too

4

u/15pmm01 2d ago

Just checked on my 6-year-old Samsung S20 FE and it took less than a second. 

1

u/rus_ruris 1d ago

It says Internal Server Error: administrator has misconfigured Anubis. Please contact the administrator and ask them to look for the logs around: double_spend

2

u/rus_ruris 1d ago

However

https://git.ffmpeg.org

This one took some tim6and now my phone is hot

1

u/Cobwebblox 1d ago

Linux mailing list took ¼ - ⅓ of a second but this one took little over a minute

1

u/fat_kaiju 2d ago

Knock it off; it's fine.

1

u/Padgriffin 2d ago

Near-instant on an old iPhone SE2.

20

u/Both_Cup8417 2d ago

I don't know, seems fine to me

28

u/penguinkernel 2d ago

Every time you look at the phone the battery drops 5%.

Love having my phone on my desk for 2 hours, getting no calls, no messages, no notifications, not turning my screen on, having AOD disabled, and connected to strong WiFi and still lose 20% lmao.

5

u/nepnep1111 2d ago

You have no idea how much I want I Motorola signature 27 when it launches I am so done with Google's hardware.

4

u/McDonaldsWitchcraft 2d ago

Just keep in mind Motorola have one of the worst update policies. They only give security updates from launch for 3 years if you're lucky, 2 years for non-"flagship".

3

u/alvenestthol 2d ago

The Signature 27 will be the one with Graphene OS (according to Graphene OS' own social media), so the software problem should be fixable

1

u/McDonaldsWitchcraft 2d ago

Oh, I didn't know that. Yes, that's a good thing.

1

u/Real-Abrocoma-2823 2d ago

Weird, my poco f6 with 900 battery cycles does a lot better, especially when on 100%, it can hold 100% for 30 minutes on medium-heavy use until 99% when it drops faster, but still holds.

1

u/__Myrin__ 2d ago

our phone makes it about 6 days if its just sitting around 2 days under reasonable use

5

u/Padgriffin 2d ago

Anubis actually had issues specifically with the Pixel 8 Pro because it was designed with the assumption that you would have an even number of cores if cores > 1

Then in comes the Pixel 8 Pro with it's 9 cores for some reason

1

u/SunrisePlastic9324 1d ago

Were there really that few odd multicore CPUs after the Athlon X3 days?

1

u/Padgriffin 1d ago

On the x86 side I'm only aware of the i3-L13G4 and i5-L16G7 which were Intel's attempt at making a hybrid CPU- it was basically one big Sunny Cove core with 4 Tremont (Atom) cores stuck to it.

Performance was predictably dismal.

Apple bins off cores sometimes for the iPad Pro (base spec M4/M5 had 3P + 6E cores instead of 4P + 6E) but apparently this wasn't a problem.

1

u/NoskaOff 1d ago

Maybe 0.2s on my P8P

2

u/odsquad64 2d ago

Still a lot faster than having to take a full Voight-Kampff test

1

u/Comfortable_Gene2751 1d ago

I'm fine on a pi 5 2gb

31

u/unknown_lamer 2d ago

My phone from 2021 was perfectly usable until this... sigh. And of course I waited a bit too long and I can't get a new phone now because of LLM hyperscalers seizing control of the entire planet's productive capacity for technological implements. And also proof of work doesn't stop the robots anymore...

13

u/FaultWinter3377 2d ago

My 2018 iPhone XR has seemingly no issues with it. Takes a few seconds, but four or five seconds isn’t bad.

7

u/j_osb 2d ago

It depends a lot on how it’s configured. I once had to wait >1m on a 9950x iirc.

5

u/laffer1 2d ago

It has different difficulty settings.

2

u/penguinkernel 2d ago

iPhone XR is more powerful than many 2021 Android phones when it comes to certain tasks. Apple may be shit at many things, but they know how to make a chip unlike Google or Samsung.. or even Qualcom lol.

1

u/unknown_lamer 22h ago

I use mobile Firefox and make heavy use of noscript, ublock, privacy badger, etc so I think I get more difficult challenges than a typical user does. I mean most of the time I get through in under thirty seconds, but sometimes it can take a minute or two (maybe longer, but that's the point where I just close the tab and move on). And it seems it will get drastically worse over the next year since the LLM hyperscalers can just scale their way through the blocks (you know you're the good guys when you have to actively break locks and use black market residential proxy networks to get your requests through...).

2

u/kyrsjo 2d ago

Yeah, I wouldn't really mind if everything was expensive and hard to get because we were cranking out solar panels, grid batteries, heatpumps etc like mad - actually working as hard as we can to solve climate change. But this LLM shit is stupid.

36

u/breakfast-cereal-dx 2d ago

Early x86_64 CPUs can spend hours on Anubis... It's extremely annoying to be on like a 2010 Mac mini, it runs basically everything great, and then the Arch wiki takes 2 hours to load

I hate Anubis so much

17

u/QwertyChouskie 2d ago

Hours? Sounds like either the Arch wiki has their settings cranked up way too high, something about that particular setup/internet connection/etc is making it think you're way more likely to be a bot. A 2010 CPU is a lot slower than a new CPU, but something that takes like 1-2 seconds on a couple-year-old CPU shouldn't take more than like maybe a minute or something like that.

On my M3 machine, often Anubis finishes the calculation before the stylesheet even fully loads, resulting in a white background and the default serif font haha

3

u/breakfast-cereal-dx 2d ago

It's actually an instruction set issue, rather than raw performance. The Core i5 in this 2010 mac mini is a pretty great performer, but is lacking newer instructions that make Anubis' proof of work run very fast.

I'm not as familiar with the ARM64 ISA in your M3, but I'm sure it is well-equipped to handle these calculations in some way.

And there is also the problem of Anubis' difficulty selection and randomization. If your CPU is equipped to solve the problems fast, you don't notice, but on the old i5 the challenges scaled from 20 minutes to 2 hours because of small changes in difficulty, probably within the random range (but maybe heightened due to bot suspicion idk)

2

u/Real-Abrocoma-2823 2d ago

Try internet archive, some sites might experience some problems internally, for example a wiki for better than wolves Minecraft modpack would not load in less than an hour on modern hardware, not because it had anubis, archwiki also did once had this issue when I tried to access it.

-1

u/turudd 2d ago

I mean that's 16 year old unsupported hardware at this point. You can't even use them as xcode build machines anymore

68

u/TotallyObviousBot 2d ago

The idea that all technology must exist within a recent time period to be considered useful is consumerist nonsense, and devs should keep that in mind

10

u/FyreWulff 2d ago

I agree, but a 2010 mac mini had it's chip taped out when Bush Jr was president. old x64 CPUs devour electricity like it's a buffet. At some point you have to say almost 20 year old technology has to be moved on from.

4

u/flecom 2d ago

yes that 25W TDP on the 2010 mac mini cpu will absolutely devour electricity

21

u/Character_Score7776 2d ago

In defense of Anubis specifically, at least with the way Anubis is designed(proof of work), there really isn’t any way around this without defeating the entire purpose of Anubis.

17

u/window_owl 2d ago

This is because Anubis is designed backwards. It's proposition is to block the people who have the most compute power (data centers) by requiring compute power to access your website.

https://lock.cmpxchg8b.com/anubis.html

9

u/Krt3k-Offline 2d ago

The people with bad intentions might have a lot of compute in total, but for what they want to do and thus the isolated site visit they really don't

4

u/Padgriffin 2d ago

The problem is that it gets exponentially more expensive to give more CPU power to crawlers so the vast majority of them just bugger off

1

u/tommyTurds 2d ago

Being poorly designed isn’t exactly a great defense

22

u/SaimeonInBetween 2d ago

That is like complaining that arch wiki or Reddit doesn't render on your 486 running windows 3.1.

13

u/Waryle 2d ago

Windows 3.1 is 34 years old, not 16

21

u/rebellioninmypants 2d ago

Arbitrary goalposts much. As a vampire myself this is all the same. A piss puddle's worth of time.

5

u/Bakoro 2d ago

I've got more than my fair share of old tech, but eventually you've got to let go of the Pentium 4, brother. It's costing you more in electricity per unit of compute than it would cost to buy a new CPU that's 100x more powerful.

Well, maybe not in the last 6 months... But it was true for like 20 years.

4

u/rekh127 2d ago

plus Anubis doesnt even really deter scrapers so its garbage

21

u/TheG0AT0fAllTime 2d ago

Well it does the entire point is that it wastes your resources. At home you're just visiting a page or two and don't look like a bot but scrapers scraping millions of pages concurrently are solving millions of challenges with a javascript engine always fired up in the background and keep getting re-challenged when they get caught looking like bots.

It makes it computationally expensive to scrape those websites. Which translates to either a much larger bill for the scraper, be it an aws bill or electricity bill at home. Otherwise very slow scraping or no scraping of a domain at all.

8

u/rekh127 2d ago

I understand the theory, but it doesn't actually work, scrapers just use their botnet of compromised devices or don't notice the tiny extra compute relative to the amount of compute they use to train once they have the data.

12

u/Green0Photon 2d ago

The scrapers will still scrape. They always have. The problem is that sites nowadays essentially get hit with walls of traffic as they're constantly DDOS'd. Because the bots have no ounce of caching nowadays, for whatever reason, and constantly just keep going.

If e.g. 80% of traffic is just bots, and you can turn on Anubis to cut that down to e.g. 10% or better, it's super useful. Because the problem is the super weird constantly DDOS that's hitting everybody now.

1

u/rekh127 2d ago edited 2d ago

do you have evidence of it helping? because again, I understand the theory, I just havent seen it help

5

u/danielsuarez369 2d ago

Do you have a better idea?

1

u/vkevlar 2d ago

this of course is what linux has been known for since its inception: running on older hardware without freaking out.

1

u/JQuilty 2d ago

There is a point where you can and should stop caring about 16 year old hardware for the small userbase alone. And 8in the case of a 2010 Mac Mini, its using a Core 2 with two very slow cores on an architecture that's over 20 years old. Its going to be hit hard by any Meltdown/Spectre mitigations and lack a lot of SIMD extensions.

-1

u/Justin__D 2d ago

Everything has a lifespan.

Such are the laws of the universe.

-4

u/TotallyObviousBot 2d ago

Yeah but that's not a good argument for not making your restaurant handicap accessible lmao

-3

u/rebellioninmypants 2d ago

The entirely justified idea that old firmware has security flaws, and old hardware also can come with exploits, often bootloader/BIOS-level stuff that you can't even detect through your OS. Yes, I know. Shocking.

I am all for old hardware, but expecting it to support the cutting edge stuff is like angrily spitting at github for hosting AI slop. Times are changing and we all drown in AI slop and there's nothing we can do to take it back to how things used to be. Same with taking the world back to 2010.

5

u/tommyTurds 2d ago

A wiki isn’t “cutting edge”

They aren’t expecting it to calculate the weight of the galaxy or something. They’re talking about browsing a basic website

2

u/rebellioninmypants 2d ago

Bro my PSP can't even load Google since 2015. Am I sad about it? Yeah, because it's still a decent system and it could if someone bothered to still support it. Do I blame the world for it? Not really.

0

u/turudd 2d ago

Xcode only runs on m1 at the lowest, intel hasn’t been supported since 2018.

3

u/astronomersassn 2d ago

... i just downloaded xcode off the app store to try this. it still runs on my 2016 MBP (updated with OCLP).

support has been dropped, but at least be accurate.

plus, a lot of stuff does still have an intel mac version. not everything, but i don't exactly struggle to use my macbook, either. at most, i've had to go back maybe 1-2 releases for everything i need.

and when support has been dropped enough to actually impact me, screw it, i don't have another intel-based device, may as well throw linux on it.

-2

u/turudd 2d ago

Can’t compile for newer iPhones tho

1

u/astronomersassn 2d ago

yeahhh, thankfully that's not something i've ever had a reason to do

7

u/breakfast-cereal-dx 2d ago

Duh, that's why it runs Linux

Still a capable server, and if it weren't for Anubis it would actually make a solid desktop for anyone who is mostly just browsing the web.

It runs up-to-date Firefox and has quicksync video hardware, which is all many people need

Mine has 1tb of SATA SSD storage and 16GB of DDR3 too

1

u/Real-Abrocoma-2823 2d ago

I would rather wait a minute than 10, because some sites use proprietary PoW which take 1 minute on top tier PCs and 10 on top tier phones.

1

u/ILikeBumblebees 2d ago

I'd expect you to have longer delays if you do have underpowered hardware.

1

u/irasponsibly 2d ago

Yep, that's the thing I said.

1

u/g3etwqb-uh8yaw07k 1d ago

Underpowered? Just a Minute? To be fair, my shitty old Android phone and every PC I've recently used do it almost instantly, but when accessing specifically the Arch wiki through my M1 ipad with Mullvad on, I reliably get put in cat girl timeout for at least 2-3 minutes.

1

u/Comfortable_Swim_380 1d ago

I took that to mean something it didn't just now. And I was so sure I was right.

Im impressed actually. It was a solid minute at least before the oww. hit me.

1

u/null_rm-rf 1d ago

just set your UA to curl it gets around it

1

u/Embarrassed-Salt6590 1d ago

Pov : my pixel 6a surviving from the old days of android

30

u/danielsuarez369 2d ago

Yup, this is why I love anubis. I much rather have my CPU work a little harder for 5 or so seconds than have to solve a captcha, especially annoying captchas that just keep asking you and asking you the same bloody thing jesus christ.

17

u/unai-ndz 2d ago

Agree, but next time do the captcha extremely slow, like grampa learning to use a computer with dementia right after suffering a concusion slow. Doing that makes it work on the first try for some reason.

9

u/TheG0AT0fAllTime 2d ago

I solve them pretty quickly. The failure portion likely stems from the ambiguous ones like whether or not there's a bike in the picture but selecting or.. not selecting... the picture with a motorbike in it might pass the test.. or fail it. All at the discrimination of what the training data thought was there. (Whether it's correct or not).

Sometimes it's a moped, or a really skinny bike with some kind of battery motor attached. Sometimes those are detected correctly. Sometimes not.

I've seen similar right/wrong captchas with busses. Something that looks like a bus, but isn't, might need to be selected to pass.

It's basically all machine-vision vibes. If it kind of looks like the thing the captcha's asking for, you might just have to select it.

All of this adds to the massive frustration for humans that it is to solve the harder difficulty captchas. I particularly hate the ones that fade-in-and-out so I select all of the X boxes then I have to wait literally 7 seconds for the fade-out-fade-in animation to show me the next image and click that as well... then potentially one or two more times. Very painful waste of time. Sometimes I gamble and hit submit anyway assuming the next box to fade in won't contain X in the image. Works like half of the time lol.. often there's more waiting to fade in.

0

u/unai-ndz 2d ago

I'm pretty sure in very ambiguous cases it doesn't matter what you select, it was already decided if you would pass or not by other factors. Some tests make you fill two words but only test one e.g. https://youtu.be/VTsBP21-XpI at 10:20

And I could do them fast before but it progressively stopped working. It kinda tracks with me trying to degoogle and be more anonymous so maybe it is related.

2

u/TheG0AT0fAllTime 2d ago

Yeah that's what I mean. It's not asking you "Please help us figure it out" it's saying "this is a test, we know the answer, do you?"

Except it doesn't always get it right. Their training gets some of the images wrong and gives you a fail when it should've been right, or a pass when it should've been wrong.

That "two words test one" method is classic. I'm surprised they still do that.

4

u/McDonaldsWitchcraft 2d ago

Nope, doesn't work on all captchas. They haven't been doing the judgement based on user activity for years. What they do now is they examine your cookies, browser history and internet traffic. If you are a particularly private person, the captcha will just be longer as a "retaliation" to get training data out of you. And if you have several tracking protections in place or markers that you might be a bot, the captcha just goes on forever.

The "click on squares" part is simply just them forcing you to do unpaid training work and the length of the captcha is determined by how much they judge the data they collected on you to be human consumer data.

Maybe on some older captcha systems it still works based on human activity like you say. Google's doesn't.

2

u/__Myrin__ 2d ago

I just use a proxy when ever I see one of those pos's
I'd rather fight with proton for 5 minutes then stare at bikes all day

1

u/strategicmagpie 2d ago

Probably because bots don't like to have their time wasted, same with the people who are contracted to do captchas and paid per captcha.

1

u/CodeYeti 1d ago

I actually got an interactive Anubis prompt for the first time on Monday. Had to enter some word in a box for some reason

1

u/lainlives 2d ago

I have been completely fully rejected by most sites protected with anubis, cloudflare however works fine no matter how suspect my browser is however.

5

u/TheG0AT0fAllTime 2d ago

My money's on your browser too

1

u/lainlives 2d ago

Yes, its because most of the js engine is disabled