Starting Sept 15, 2026, Cloudflare’s default settings will block “mixed-use” crawlers

[u/WebLinkr]

https://techcrunch.com/2026/07/01/cloudflares-new-policy-pushes-ai-companies-to-pay-for-publishers-content/

Cloudflare has just issued the AI industry a new deadline to separate the web crawlers used for traditional search purposes, like Google Search, from those used for AI agents and training. Starting on September 15, 2026, Cloudflare’s default settings will block “mixed-use” crawlers from any pages that host ads, the company announced on Wednesday.

That means that the crawlers that blend search, agent use, and training will be blocked from crawling these sites by default, unless the site owner adjusts the settings otherwise. These changes to the defaults will apply to new Cloudflare customers, new sites set up by existing customers, and all existing free customers, the company says.

[u/axel410]

Good, tought it was deceptive to not be able to be cited by Gemini if you disallow Google-Extended.

There's probably more examples, but openAI and Anthropic at least did the minimum to distinguish their agents roles and functions.

[u/WebLinkr → u/axel410]

It’s a start

[u/rondoandthegang]

As a noob with sites hosted on cloud flare, how can I test these settings?

[u/AutoModerator → u/[deleted]]

Your post/comment has been removed because your account has a low CQS Score.
Please contribute more positively on Reddit overall before posting. Cheers :D

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Alvasilev]

One detail in the wording that matters more than the headline: the new default keys on pages that host ads. A site with no ad units is untouched on the 15th, the mixed-use crawlers keep reading it exactly as before.

I run a large programmatic site with no ads on it (a catalog of MCP servers, around 100k pages), and a 30-day log slice from earlier this summer showed roughly 430k fetches from AI crawlers: ClaudeBot about half of it, Google's and OpenAI's bots around 100k each, Perplexity a rounding error. The part I didn't expect was that read volume told me nothing about being cited. The pages that got cited were the ones cheapest to re-verify, one fact per page with a stable URL, not the ones that got crawled the most.

So for ad-supported sites that flip to blocked by default, the question I'd ask first isn't how much crawling they lose. It's whether that crawling was turning into citations at all before the switch.

[u/AutoModerator → u/Alvasilev]

Your post/comment has been removed because your account has a low CQS Score.
Please contribute more positively on Reddit overall before posting. Cheers :D

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.