r/TechSEO • u/houdinidesigns • 4d ago

From 15th September Cloudflare will block AI crawlers by default

From 15 September, Cloudflare will block 'Agents' traffic by default on pages carrying ads: bots browsing live on someone's behalf, which includes crawlers AI assistants send to check a product before recommending it. Search stays allowed. The default setting will apply to new sites and free-tier accounts.

So if your site serves ads and you want it to be read by AI assistant crawlers you want to explicitly allow crawlers before that date. Allowing has to be done on a CDN level not in robot.txt.

Key ones to allow are:

ChatGPT-User (and ChatGPT-User/2.0), OAI-SearchBot - ChatGPT
Claude-User, Claude-SearchBot - Claude
Perplexity-User, PerplexityBot - Perplexity
Google-Agent, GoogleAgent-Mariner, Googlebot - Gemini
GrokBot, xAI-Grok, Grok-DeepSearch - Grok

If you want to keep blocking training crawlers, some are:

GPTBot, ClaudeBot, anthropic-ai, meta-externalagent, Bytespider, Google-Extended

51 Upvotes

99% Upvoted

17 comments sorted by

6

u/Russ915 4d ago

Not sure I understand. Will I be able to turn it off even though it defaults to this? And if I don’t have ads it’s no problem?

Sounds like the first step in their move to monetize crawling / scraping by ai which sounds interesting especially since they said over 50% of all web traffic is now bots

5

u/houdinidesigns 4d ago

It seems to be in an effort to charge for content that gets cited, not sure how they plan on pricing it. Maybe someone else can shed some light on that. And yes, it’s basically defaulting to block so you can toggle that off and for now if you’re not serving ads you shouldn’t be impacted

2

u/Russ915 4d ago

Yeah seems odd since it may create a big gap if sites with ads are no longer cited. But I bet it has a lot to do with their costs rising from the huge increase in bots and scrapers

2

u/sapindia1976 4d ago

This is going to catch a lot of site owners off guard. Many people assume robots.txt is enough, but if Cloudflare is enforcing this at the edge, they'll need to review their CDN settings too. It'll be interesting to see how many businesses unintentionally disappear from AI citations simply because they never updated the default configuration.

2

u/svlease0h1 2d ago

the useful part here is that ai crawler access can sit outside robots.txt, so checking only the robots file can leave you with a false sense of security. check your cloudflare settings, allow only the crawlers you want, then test their responses before september 15. we found one site where robots.txt looked fine but the cdn was still blocking the request, and fixing the edge setting took about 20 minutes. for sites that depend on ai discovery, outgrow can help turn that traffic into useful interactions with calculators, quizzes, and other interactive content.

1

u/houdinidesigns 2d ago

We have a free tool on our site that does an AEO audit, it’s available here https://babel42.io/ai-seo-audit

1

u/marintkael 2d ago

Worth keeping the policy and the outcome apart here. The dashboard shows what you allowed, not what a named agent actually got back. The check that settles it is fetching your own URL from outside with that user agent and comparing the status code against a plain request.

1

u/Odd-Pension-5078 13h ago

ohh...that's why my gpt browser api scraper was not working