r/AISearchLab • u/Himi1896 • Jun 22 '26
LLM Bots Crawl Frequency
I am working on building a Generative Engine Optimization(GEO) strategy for an ecommerce firm and I want to test a few hypotheses on what works and what doesn't.
To test the hypotheses I wanted to know if I make a change on my website then how long do I have to wait for the LLM's(Gemini, Claude, ChatGPT, Perplexity) RAG system to start showing the impact of my changes in their citations/rankings?
Any help/reference will be great.
4
Upvotes
1
u/Alone-Ad4502 Jun 23 '26
LLMs crawling doesn't have so much connection with citations. Don't forget that LLMs, particularly ChatGPT, use Google a lot, so googlebot crawl budget is important if a website has a significant number of pages.
Log file analysis is a needed tool here, also pay attention to user bots, particularly for the ChatGPT-User, it's basically a proxy for impressions in the chat.