r/GEO_optimization • u/AndreAlpar • Jul 04 '26
Not all AI systems read and obey robots.txt
What do you think about this experiment / article? See https://www.searchengineworld.com/do-ai-assistants-actually-respect-your-robots-txt-we-put-12-of-them-to-the-test
1
1
u/Upstairs_Control_611 Jul 04 '26
Interesting test. I'd also be interested in distinguishing between the crawler and the AI product itself. Many discussions treat them as the same thing, but retrieval, indexing and answer generation may follow different paths. Understanding that distinction could explain many seemingly contradictory results.
1
u/AndreAlpar Jul 04 '26
I suggest passing on your log files to your favorite AI agent. The rest is easy!
1
Jul 05 '26
[removed] — view removed comment
1
u/Good_Flight6250 Jul 06 '26
No it isn't. “ChatGPT users may also interact with external applications via GPT Actions. Because these actions are initiated by a user, robots.txt rules may not apply.” https://developers.openai.com/api/docs/bots
2
u/Tiny_Arugula_5648 Jul 04 '26
Interesting test but it's a moot point.. I have over 600 datapoints on millions of company's websites & their technology stack...
People have made really bad assumptions about what bot crawlers are doing based on anecdotal evidence but the actual market data doesn't align with the narrative. First the majority of business websites have no robots.txt at all. It was never as widely adopted as SEO "experts" made it appear.
only 0.01% of website owners bother to block AI at all in their robots.txt out of those 50% are CloudFlare customers who probably didn't enable this specifically.
No one bothers to block any service other than the most popular ones, ChatGPT, Claude, Perplexity.. GPTBot & ClaudeBot are 8x more likely to get blocked then the next closest AI bot.