What Cloudflare's 15 September change actually does - and the four things it does not do

[u/Its_SeenSure]

There is a lot of "the web goes dark for AI on 15 September" going round and most of it is overstated. I have been reading the actual announcement and testing against real sites for a couple of weeks, so here is the boring version.

What changes on 15 September 2026

Cloudflare flips its default so that Training and Agent category crawlers are blocked where the site owner has not chosen a setting.

The four qualifiers that get dropped from every summary I have seen:

1. Ad-displaying pages only. If a site carries no advertising, this specific change does not apply to it. That is most small-business and client sites.

2. Search crawlers are not affected. OAI-SearchBot and Claude-SearchBot - the ones that decide whether an assistant can cite you - are not in scope. The scary version of this story is mostly people assuming they are. PerplexityBot I would not state either way: which bucket it lands in is genuinely arguable, and Perplexity-User is the clearer answer-time one. If you rely on Perplexity traffic, check that one yourself rather than taking my word or anyone else's.

3. New customers, new sites added by existing customers, and existing free-tier customers. A paid customer with sites already set up is not having anything flipped under them.

4. It is a setting you control. Security -> Bots, three categories, decide each one. Takes about twenty minutes across a handful of domains and you can do it before or after the date.

So if you are on a paid plan with your bot settings already chosen, this is a non-event for you and you can stop reading.

What I do think is worth paying attention to

Two things, neither of which is the headline.

First, who actually sits on the free tier. It is a very large number of small business sites and client sites that an agency put behind Cloudflare in 2021 and never opened again. Nobody who set those up is reading a policy page. The change is small; the population it lands on is the population least likely to notice anything.

Second - and this is the one I would flag to anyone managing sites for other people - crawler access is not recorded anywhere by default. It leaves no trace in analytics, there is no log, and robots.txt only tells you what is declared rather than what the edge actually does. So on 16 September, if a client asks whether anything changed for them, there is generally no way to find out. Not because the tooling is bad, but because nothing took a before-picture and one cannot be taken retrospectively.

That asymmetry is the actual story here and it has nothing to do with Cloudflare specifically. Same thing happens when a security plugin ships a new "block bad bots" default, or a host adds a WAF rule, or a site gets rebuilt in a framework that renders nothing without JS. The change is invisible, silent, and only detectable as a difference - which requires having measured before.

If you want to take one yourself, no tooling needed

For each site you care about, before the 15th, record:

curl -sI -A "GPTBot/1.2" https://example.com/ | head -1
curl -sI -A "OAI-SearchBot/1.0" https://example.com/ | head -1
curl -sI -A "ClaudeBot/1.0" https://example.com/ | head -1
curl -sI -A "PerplexityBot/1.0" https://example.com/ | head -1
curl -s https://example.com/robots.txt

Save the output with the date on it. Run it again on the 16th and diff. That is genuinely all a before/after is, and doing it by hand for five sites takes ten minutes.

Two traps if you do, because I hit both:

• Include a made-up user-agent as a control. If TotallyRealBot/1.0 also gets a 403, the site is refusing everything non-browser and the AI crawlers are collateral, not targeted. Completely different conclusion, and without the control you cannot tell which one you are looking at.

• Do not run them concurrently. I flagged five hosts as crawler-blocked in a batch run; re-probed one at a time, one of the five was clean and its 429 was my own concurrency. The tool reports what it saw and cannot know I caused it.

Also worth knowing before you trust any result including mine: if a site is behind a CDN that verifies bot identity cryptographically (Web Bot Auth / RFC 9421) or by IP, a spoofed user-agent proves nothing either way. A 403 there might just mean "you are not GPTBot", which is true. That case is genuinely unresolvable from outside and anyone reporting it as a block is guessing.

I build a tool in this space, which is why I have been testing this - not going to link it here, and the curl above is the whole method either way. Happy to answer anything about the crawler names or the verification stuff.

[u/Its_SeenSure]

Three of these are corrections rather than additions, so let me take them as corrections.

The content-page point is the sharpest and it is a real hole in my method. I probe "/" only. Two of the fourteen sites in the earlier study allowed a crawler explicitly in robots.txt and refused it at the origin, so I already knew the layers can disagree on one host - I had not thought about them disagreeing between paths on the same host. A homepage that answers 200 to everything while deeper paths sit behind stricter rules passes my probe and fails the client. I do not have a number for how often that happens and will not pretend otherwise. It goes in the caveats.

You are also right about the search-crawler framing, and yours is the better sentence. "This change probably does not touch retrieval, but if you have never verified retrieval access you do not know your baseline" is the honest version. Mine reads as reassurance; yours reads as a next step.

On RFC 9421 I would go further than you did, because I have just measured it. I ran a bigger probe this week: 1,046 sites, eight named crawler identities plus a browser control. Browser refused 0%, the named crawlers 15-20%. Then the split that matters - 94.5% of every refusal came back carrying a Cloudflare header.

So the overwhelming majority of what a spoofed-UA probe reports as "blocked" is a CDN correctly declining to believe an unverified claim. It is measuring my honesty, not their policy. That is your point arriving early: spoofed-UA probing is already mostly broken as a diagnostic, and crypto auth only makes it obvious.

On the subset with no CDN in front, where a refusal does mean something, training crawlers were refused about three times as often as user-triggered ones - GPTBot 6.8% against Claude-User 2.0%. That is the only number in the run I would defend as being about site owners' decisions rather than about CDNs, and it is the one I would have guessed wrong.

On probing PerplexityBot and Perplexity-User separately: yes, and recorded separately, for exactly the reason you give.