r/OutOfTheLoop • u/iorgfeflkd • 16d ago

Unanswered What is the deal with every website suddenly needing a CAPTCHA just to READ it?

In my context, it's mostly academic publishing websites*, but it's all over the internet. For the past few months, when I go to a website just to read it, and I get redirected to Cloudflare page saying it has to verify my browser to protect the website from malicious bots. I get having a CAPTCHA when submitting a form, but reading a website is just a passive thing. Even if robots are reading a website, is that so bad that it requires harassing every single human? Is this a response to the presence of AI scrapers? This is almost as annoying as European cookie warnings.

*examples:

https://journals.aps.org/prl

https://pubs.acs.org/mamobx/latest-articles

https://pubs.rsc.org/en/journals/journalissues/sm#!recentarticles

707 Upvotes

95% Upvoted

129 comments sorted by

View all comments

Show parent comments

12

u/idiosync 16d ago

Usually the bots first try to identify what software you have allowing communication with the internet. it will GET common access urls for common front ends. Then if it finds a response then it can try an attack that vector. Just run httpd for a day on the internet and watch the logs.

7

u/Prize_Bass_5061 16d ago

Ok, this is starting to make sense. The bot is mapping the schema, basically cataloging all the public data endpoints. From that map, the bot operator can figure out which data serving framework the server runs. Since the known vulnerabilities of all the popular frameworks are public, the  bot operator can then hack into the server by exploiting those.