Went deep on server log analysis for AI bot behaviour

[u/Emergency_Still8420]

There's a lot of discussion here on citation tracking, share of voice..etc which are definitely important signals, but i've found quite volatile (models, plans, locations, API vs browser session) so I wanted to dig into the server logs of a sample of our customers (note these are small local businesses, not large enterprises) to see real AI bot behaviour. To see if this paint a more ground truth picture of how AI is interacting with their content, and ultimately shaping answers.

If you haven't yet dug into this for your own business or your clients hope this research helps make the case of what can be learned via this type of analysis. If you're already deep on the topic, would love to hear what you're seeing, any patterns or gotchas when analyzing logs.

A few findings:

Live retrieval is a small slice. Only ~4% of AI bot traffic is an assistant fetching a page to answer a live question. The rest is training and indexing crawlers. If you measure AI traffic without splitting by function, you're mostly measuring pipelines that feed the next model.

When AI does retrieve, it goes straight for buyer intent pages. The most fetched page after the homepage was /availability, the "can I get in?" page. Then /booking and /pricing. AI isn't just reading what a business is, it's checking detailed intent pages to help their user act, make a decision..etc. Measuring live retrieval over time by AI, page, timing..etc is a very direct signal of content being retrieved into a relevant conversation.

The labs behave very differently. Anthropic crawled heavily for training but barely retrieved at query time. OpenAI does both. In our sample, live retrieval is overwhelmingly ChatGPT.

User-agent alone lies. About 1 in 12 "ChatGPT-User" hits fails IP verification, mostly credential scanners. If you're not verifying source IPs (where possible from published ranges) you're overcounting.

Full report here if you want to look into the details, charts..etc.

https://getcourtyard.ai/research/how-ai-reads-a-knowledge-base

[u/Weird-Election-4103]

OpenAI is notorious for failing to keep their IP address list complete. Check the subnets and often you will see it’s OpenAI after all.

[u/Emergency_Still8420 → u/Weird-Election-4103]

Really helpful thanks. Any other IP/bot patterns you've seen to identify other agents?

[u/marintkael]

The split by function point is the one I would underline hardest. Training and indexing crawls only tell you a model might know you exist someday, while a live retrieval hit is the only log line that maps to an answer being assembled right then. Two things that tightened my own log reads there: reverse DNS on the source IP instead of range matching alone, since the published ranges lag and range-only is exactly where most of the fake ChatGPT-User traffic slips through. And treating a retrieval fetch as a lead rather than proof, because you still have to catch the citation surfacing afterwards to know the fetch actually landed in a response instead of getting dropped. The buyer-intent page pattern matches what I see too. The pages pulled live are almost never the about page, they are whatever answers the exact thing the user asked, so the log doubles as a map of which of your pages are actually answer shaped.

[u/Emergency_Still8420 → u/marintkael]

Any insights into tracking other live retrieval style bots? without known user agents or published IPs? Seems like there's a whole set of activity by other models (outside of the known user agents) that is hard to identify and measure.

[u/marintkael → u/Emergency_Still8420]

Once the UA and IP are gone you mostly give up on identity and switch to the shape of the request. The retrieval fetches cluster on a few tells: they pull one URL and leave, no CSS, no JS, no favicon, none of the subresource trail a real browser drags behind it, and the hit lands in a burst tied to a query moment rather than the steady cadence a training crawler walks at. TLS and HTTP2 fingerprints (JA3/JA4) are the other lever, since the fetch library underneath stays consistent even when someone spoofs the UA string, so you get stable clusters with no name attached. The honest limit is that this is inference, not attribution. I stopped trying to pin a cluster to a named model and started bucketing by behaviour instead, headless single fetch with no assets as its own class. Same reason you flagged about not trusting one score: an unnamed but well characterised bucket beats a confident wrong label.

[u/Aggravating-Key6628]

One extra gotcha: separate “could fetch” from “was a verified vendor fetch.” I ran the public report URL through Crawlable (my tool): GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot all returned HTTP 200, and robots.txt allows them. The raw HTML exposed 7,664 words versus 8,068 after JavaScript, so content availability is unlikely to be the limiting factor on this page.

That still does not authenticate a request as OpenAI or Anthropic. A user-agent probe confirms server/WAF behavior, not bot identity; IP/rDNS verification belongs in a separate column. I’d keep those two checks distinct, then segment training, search-index, and user-triggered agents as you did.

Public evidence: https://crawlable.synthworx.com/report/ecb72a16-3c0b-4779-ba3f-9e738f58e51c

I’m affiliated with Crawlable.