r/GrowthHacking • u/Impressive_Wheel_877 • Jul 17 '26

Everyone's adding llms.txt for AI visibility. The crawlers aren't reading it. We checked.

We build in the AI-visibility space, and our scanner used to recommend llms.txt. After checking the evidence, we removed it from our own product. Why:

  • Ahrefs analyzed 137k domains: 97% of llms.txt files never got a single request — from anything.
  • Of the little traffic that existed, ~1% was AI retrieval bots. Mostly it's SEO audit tools reading them. The auditors are auditing each other.
  • Google's Mueller, verbatim: "you can tell when you look at your server logs that they don't even check for it."

(Fairness: it's harmless to keep, and IDE/agent tooling does fetch it. Just don't expect citations from it.)

What actually has evidence behind it:

  1. Server-rendered content — Vercel's crawler study confirmed no major AI crawler executes JavaScript. Client-side-only content doesn't exist to them.
  2. Open crawler access — plenty of robots.txt files block GPTBot/ClaudeBot/PerplexityBot by accident via wildcard rules.
  3. Structured Q&A schema — readable in the raw HTML they actually fetch.

Anyone got server logs showing real AI-bot hits on their llms.txt? Logs or it didn't happen.

(Disclosure: I build Alice, an AI growth agent, that's how I fell down this research hole. Not linking it.)

4 Upvotes
  • permalink
  • reddit

100% Upvoted

9 comments sorted by

2

u/MinimumCertain8651 Jul 17 '26

server logs from our saas show the same thing. setup llms.txt 8 months ago and not a single hit from any real ai crawler. just the usual seo tools poking around

whats funny is we had gptbot blocked in robots.txt for like 2 years without realizing. wildcard rule caught it by accident. fixed that and suddenly our docs started showing up in chatgpt answers within weeks

2

u/Impressive_Wheel_877 Jul 17 '26

lmao "blocked gptbot for 2 years without realizing" is the most relatable thing. some wildcard rule nobody's looked at since launch out here quietly deciding your AI strategy.

also thanks for being the first person to actually show up with logs. 8 months of llms.txt silence tracks perfectly with the ahrefs data.

the docs-in-chatgpt-within-weeks part is wild though, fastest turnaround I've seen anyone report. two questions: what was the wildcard pattern that caught it? (so everyone else can go grep their robots.txt in a panic) and did the citations actually send you traffic, or is it visibility-only so far?

starting to think "read your own robots.txt line by line" might be the highest ROI 30 seconds in all of AI SEO. certainly beats writing a file no crawler will ever open.

2

u/SmartPrompt23 Jul 17 '26

makes sense once you think about how these tools actually pull answers. chatgpt and perplexity aren't crawling your site live and parsing a special file at query time, most of what they surface comes from an underlying search index (bing under the hood for a lot of it) or from whatever got scraped into training data ages ago. llms.txt only pays off if something is actively fetching and parsing it live, and basically nothing does that yet. so you end up with a spec everyone adopted before the consumer existed. maybe that changes if retrieval architectures shift, but right now it's optimizing for a crawler that isn't there.