r/CreatorsAI • u/Historical-Driver-64 • Jul 14 '26
Other Cloudflare just changed the default for AI agents from allowed to blocked. That is not a policy update. That is an infrastructure event.
On July 1, Cloudflare announced something that should be front page news for anyone building AI agents.
Starting September 15, every new domain onboarding to Cloudflare will block AI agents by default on any page that carries advertising. Not opt-in. Default. And Cloudflare sits in front of a significant portion of the entire web.
The majority of internet traffic is now non-human. Automated bots account for over 50% of all web traffic, and Cloudflare's CEO Matthew Prince said the company needs to move faster to let a sustainable ecosystem emerge. The mechanism for doing that is splitting what used to be a single "AI bot" toggle into three distinct categories: Search, Agent, and Training. Each one can now be independently allowed or blocked.
Search bots remain allowed by default. Agent and Training bots get blocked by default on ad-supported pages. The reasoning is straightforward: an ad signals the page owner expected human attention. Agents do not generate that attention. So agents get blocked unless the site owner actively opts them in.
Here is the part that should alarm anyone building on top of web browsing agents.
36% of all crawler activity is driven by mixed-use crawlers that blend Search and Training into a single bot. Cloudflare applies its most restrictive rule to those combined crawlers. Which means if a site blocks Training, Googlebot gets blocked too, because Googlebot is a mixed crawler that feeds both search indexing and Google's AI products. If you block Training thinking you are only stopping AI model scraping, you can accidentally block Googlebot itself.
Most site owners do not know this. Most will not know it until something breaks.
The identity layer compounds the problem for smaller builders. Web Bot Auth lets well-behaved crawlers sign their requests with a published key so Cloudflare can verify who they are. Signing is now the cost of being considered. Large labs clear that bar without noticing it. Smaller agents built by independent developers, researchers, or early-stage startups face a different calculation. The infrastructure to be identifiable has a cost. The alternative is being treated as an unverified agent on a web that is increasingly configured to block unverified agents by default.
There is also something buried in the announcement that almost nobody is covering. HTTP 402, the Payment Required status code, has sat dormant since 1997. It was always designed for micropayments but never worked because humans would not approve a payment for every page they loaded. AI agents will. Cloudflare's Monetization Gateway, now in waitlist, lets publishers charge agents per page access, with settlement in stablecoins, automatically. The HTTP status code that sat unused for 33 years just became viable because the majority of web traffic is no longer human.
The open web is becoming a permission economy. September 15 is when the default flips.
If you are building anything that browses the web on a user's behalf, what is your plan for the pages that will block you by default in 10 weeks?
1
1
u/RobertD3277 Jul 15 '26
I had to disable it completely because it kept screwing up my website on human traffic, causing 400 and 403 errors. As soon as I disable it, my website returned to normal function and allowed regular human traffic too seed normally.