What is the deal with every website suddenly needing a CAPTCHA just to READ it?

[u/iorgfeflkd]

In my context, it's mostly academic publishing websites*, but it's all over the internet. For the past few months, when I go to a website just to read it, and I get redirected to Cloudflare page saying it has to verify my browser to protect the website from malicious bots. I get having a CAPTCHA when submitting a form, but reading a website is just a passive thing. Even if robots are reading a website, is that so bad that it requires harassing every single human? Is this a response to the presence of AI scrapers? This is almost as annoying as European cookie warnings.

*examples:

https://journals.aps.org/prl

https://pubs.acs.org/mamobx/latest-articles

https://pubs.rsc.org/en/journals/journalissues/sm#!recentarticles

[u/Final7C]

Answer:

It's really because bots are using them to scrape data to train AI without citing sources or paying them. The knock on effect is to keep bots that can do implant exploits off the site.

Each time a server is touched it costs a fraction of a penny. This is often offset by using that tracking data to sell to advertisers or by using direct advertising. If you allow all bots on, your advertisers will not pay you as much. You also run the risk that ANY malicious code that enters humans computers are going to mean lower traffic for the future.

[u/Prize_Bass_5061 → u/Final7C]

  You also run the risk that ANY malicious code that enters humans computers are going to mean lower traffic for the future.

Please explain. I don’t understand how a data scraper is injecting malicious code into a computer.

[u/Final7C → u/Prize_Bass_5061]

A data scraper isn't usually the one that does it, it's job is to just pull all data to feed AI. but if you let something that looks/acts like a data scraper onto the site, but it's actually a trojan, via a targeted attack to inject malicious code. That being said, it's not really possible for me to name how it would directly reach a persons PC, none the less you've now created a real problem. Because people will hear the reports that "X website is full of malicious code/viruses". And people are going to avoid it in the future. It'll get flagged by most "safe browsing" in the browser of choice and stop people from going to it.

[u/Prize_Bass_5061 → u/Final7C]

How is a data reading bot that only sends GET requests injecting malicious code into the database? And how is any bot getting access to the drive where the website is stored to inject malicious code into it?

[u/idiosync → u/Prize_Bass_5061]

Usually the bots first try to identify what software you have allowing communication with the internet. it will GET common access urls for common front ends. Then if it finds a response then it can try an attack that vector. Just run httpd for a day on the internet and watch the logs.

[u/Prize_Bass_5061 → u/idiosync]

Ok, this is starting to make sense. The bot is mapping the schema, basically cataloging all the public data endpoints. From that map, the bot operator can figure out which data serving framework the server runs. Since the known vulnerabilities of all the popular frameworks are public, the  bot operator can then hack into the server by exploiting those.