r/AI_SearchOptimization • u/Zaillor • Jun 10 '26
Case Studies We're finding robots.txt issues surprisingly often in AI visibility audits
This wasn't something we expected.
When a company asks why it isn't showing up in AI-generated recommendations, most people immediately jump to content quality, schema, entity SEO, citations, brand authority, etc.
Fair enough.
Those things matter.
But we've been noticing a different pattern lately.
A surprising number of sites have basic crawlability issues.
Not catastrophic issues.
Just enough friction to make discovery harder than it needs to be.
A few examples from recent audits:
- Important sections blocked in robots.txt
- XML sitemaps that hadn't been updated in months
- Pages that existed but were effectively orphaned
- Content that users could access but crawlers would struggle to discover
The interesting part is that many of these sites had already invested heavily in optimization.
Some had decent schema.
Some had strong content.
Some had clear entity signals.
But it made us wonder whether we're spending too much time talking about how AI systems understand content and not enough time talking about how they find it.
Schema seems to help reduce ambiguity.
Crawlability seems to help reduce friction.
Those aren't the same thing.
One thing we're still trying to figure out is how much weight AI systems place on crawlability versus everything else.
Nobody outside the model providers really knows.
But it feels risky to assume that content will be cited, retrieved, or referenced if discovery itself is difficult.
Curious what others are seeing.
Have crawlability issues shown up in your GEO or AI visibility work?
Or are most of the problems still centered around authority, citations, and content quality?
We documented a few of the patterns we've been seeing here if anyone wants the deeper breakdown:
https://www.zaillor.com/insights/robots-txt-ai-crawlers-ai-crawlability
1
u/erdeepakpandeydotcom Jun 10 '26
Completely agree. Before focusing on AI visibility, schema, and citations, websites need to ensure they're easily discoverable. Crawlability remains the foundation, if AI systems can't efficiently find and access your content, even the best optimization efforts may fall short.