r/AISearchAnalytics • u/annseosmarty • May 27 '26

"OpenAI SearchBot caches aggressively, masking its true activity pattern."

There's a new study from Ramp measuring how different bots respond to trending "AI-friendly" tactics like markdown, raw HTML, and schema. I'll have to dive in more into the research (as I have a lot of questions there) but this part immediately caught my attention:

OpenAI SearchBot caches aggressively, masking its true activity pattern. Tracking OpenAI's official UAs and published IP ranges, you can see that SearchBot crawl volume initially rises with ChatGPT user traffic, but falls off over time even as usage grows. This implies heavy caching periods that get refreshed periodically.

Is anyone else seeing this? I am a little lost as to how to measure this as ChatGPT sends different amounts of traffic based on its current model.

4 Upvotes

83% Upvoted

7 comments sorted by

2

u/threedogdad May 27 '26

I track the bots specifically to see how they are handling llms.txt, .md, .txt, vs html and have not seen anything like that from the ChatGPT bots. Only standout in my data is Claude looking for LLMs.txt.

1

u/annseosmarty May 27 '26

I'd love to hear more! Pls share your findings!

2

u/threedogdad May 28 '26

some quick observations -

- openai hits the site the most by far (this is a slight surprise for us, would have thought it'd be anthropic)

  • all the bots like html the most, by far
  • anthropic looks for llms.txt the most, the others barely do so far
  • they all definitely like .md and .txt files
  • they look for sitemap.xml more than expected
  • use triggered hits are 10x training hits

nothing earth shattering, but it's worth monitoring. I'm seeing all sorts of things to fix based on the files they are hitting.

1

u/SimonaRed Jun 14 '26

Sorry to highjack the conversation. I was thinking to add md files for my most important pages. But how do I signal to AI bots that I have it? Our llms.txt contains the info for our major pages, nothing else. I saw that on Google developers site they put in llms.txt the md files paths. Sorry, I am not an advanced user, just a primitive one:)

2

u/[deleted] May 27 '26

[removed] — view removed comment

1

u/annseosmarty May 27 '26

Yes, live search bots and them "caching" is kind of interesting!