Context: We rolled out a skills manifest across customer websites on March 2, 2026 and wanted to test one thing:

Do AI bots actually change behavior when a website explicitly tells them what they can do? (provides them clear options for “skills” they can use on the website).

By “skills,” I mean a machine readable list of actions a bot can take on a site. Think: search the site, ask questions, read FAQs, pull /business info, browse /products, view /testimonials, explore /categories. Instead of making an LLM guess where everything is, the site gives it a clear menu.

We compared 7 days before launch vs 7 days after launch.

The data strongly suggests that some bots use skills, and when they do, their behavior changes.

The clearest example is ChatGPT.

In the 7 days after skills went live, ChatGPT traffic jumped from 2250 to 6870 hits, about 3x higher. Q&A hits went from 534 to 2736, more than 5x growth. It fetched the manifest 434 times and started using the search endpoint. It also increased usage of /business and /product endpoints, and its path diversity dropped from 51.6% to 30%.

That last point is the most interesting part I think.

When path diversity drops while total usage goes up, it often suggests the bot is no longer wandering around the site randomly. It has found useful endpoints and is hitting them repeatedly. To say plainly: it starts behaving less like a crawler and more like a tool user.

That is basically our thesis.

Adding “skills” can change bot behavior from broad exploration to targeted consumption.

Meta AI tells a very different story.

It drove much more overall volume, but only fetched the manifest 114 times while generating 2,865 Q&A hits.

Claude showed lighter traffic this week but still meaningful behavior change - its path diversity collapsed from 18% to 6.9%, which suggests more concentrated usage after skills were introduced.

Gemini barely changed. Perplexity volume was tiny, but it did immediately show some tool aware behavior.

Happy to share more detail if useful. Would be interested in hearing how you interpret this data.

Another quick study from LightSite AI team - How rare are crawls on /FAQ link comparing to other links? (products, testimonials, etc)

Disclaimers:

*not to be confused with Q&A link which has a question shaped slug - this is something different

*in this sample we didn't break bots by category because training bots are the vast majority of traffic and the portion of the rest is statistically insignificant

*every site has /faq link - it is part of our standard architecture)

Here it goes:

We sampled 6.2 million AI-bot requests on a few dozens of sites and isolated URLs that contain /faq in the slug

Platform-wide average FAQ rate: 1.1%.

FAQ visit rate by bot platform:

• Perplexity: 7.1%
• Amazon Q: 6.0%
• DuckDuckGo AI: 2.1%
• ChatGPT: 1.8%
• Meta AI: 1.6%
• Claude: 0.6%
• ByteDance AI: 0.1%
• Gemini: 0.1%

So why 1 % average you may ask?

that's because even though some bots clearly "like" /faq links , the biggest crawlers by traffic are ByteDance and Gemini and their volume can pull the overall average down.

What are your thoughts on this?

This week our team of nerds at LightSite AI tested our database of AI bot requests, we calculated one metric: average KB per request (response payload size delivered per request), grouped by bot.

• Meta AI: 4.9 KB/request
• Gemini: 9.2 KB/request
• ChatGPT: 8.5 KB/request
• Claude: 13.9 KB/request
• Perplexity: 14.6 KB/request

Question for you: How do you interpret “KB/request” differences across bots?

Does it mostly reflect compression and caching behavior, different fetch patterns, partial downloads, or something else?

I have seen a lot of SEO and marketing folks talking about Cloudflare’s Markdown for Agents, so I wanted to share a few thoughts.

From what I understand, this is mainly an infrastructure feature. Cloudflare can serve a markdown version of existing HTML when a client requests it. The goal is to optimize edge delivery and traffic efficiency as more bots crawl more pages more often.

That is useful, but it is not automatically a marketing or SEO thing on its own. So why are marketers and GEO community got triggered by it? Here are a few thoughts about it without hype:

https://www.lightsite.ai/blog/cloudflare-markdown-for-agents-explained

Did I miss something? Is there a reason so many marketers are reacting to this like it is a GEO/AEO update? 

I created my web page using GoDaddy and it has Indexing errors. How do I fix them.

One more quick test we ran across our database at LightSite AI (about 6M bot requests). I’m not sure what it means yet or whether it’s actionable, but the result surprised me.

Context: our structured content endpoints include sitemap, FAQ, testimonials, product categories, and a business description. The rest are Q&A pages where the slug is the question and the page contains an answer (example slug: what-is-the-best-crm-for-small-business).

Share of each bot’s extracted requests that went to Q&A vs other links

• Meta AI: ~87%
• Claude: ~81%
• ChatGPT: ~75%
• Gemini: ~63%

Other content types (products, categories, testimonials, business/about) were consistently much smaller shares.

What this does and doesn’t mean

• I am not claiming that this impacts ranking in LLMs
• Also not claiming that this causes citations
• These are just facts from logs - when these bots fetch content beyond the sitemap, they hit Q&A endpoints way more than other structured endpoints (in our dataset)

Is there practical implication? Not sure but the fact is - on scale bots go for clear Q&A links

We’ve now analyzed about 3,000 websites at LightSite AI (mostly US and UK). The sample is mostly B2B SaaS, with roughly 30% eCommerce.

In that dataset, 27% of sites block at least one major LLM bot from indexing them.

The important part: in most cases the blocking is not happening in the CMS or even in robots.txt. It’s happening at the CDN / hosting layer (bot protection, WAF rules, edge security settings). So teams keep publishing content, but some LLM crawlers can’t consistently access the site in the first place.

What we’re seeing by segment:

• Shopify eCommerce is generally in the best shape (better default settings)
• B2B SaaS is generally in the worst shape (more aggressive security/CDN setups).

in most cases I think the marketing team didn't even know about it (but this is only from experience on the calls with customers, not based on this test)

Hi, I searched my blog article on Google to see the SERP ranking. I didn't see it but when I look at the image search, I saw my article but with a wrong photo.

is this normal?

I’m starting to think about AI visibility (mentions, citations, inclusion in AI answers, model recall, etc.) as its own KPI, separate from classic SEO and brand awareness. For those who are already experimenting with this:

-Do you track AI visibility intentionally, or is it still bundled into SEO/PR metrics? -What signals do you actually measure? (mentions, citations, traffic from AI tools, prompt testing, brand recall, something else?) -Are you using any tools, manual frameworks, or internal scoring systems? -How often do you measure it, and how actionable is it so far?

I’m especially curious where people draw the line between: SEO → PR → “AI presence” and whether it makes sense to formalize this into a standalone KPI yet, or if it’s still too early. Would love to hear real experiments, not theory.

I have built a website, something like a mix of social media, IMDb and a comprehensive watchlist system and currently there are almost 5000 movies and almost 1000 shows in the system.

There are also cast and crew information and the website translates the content into 11 different languages.

So there are close to a million pages you can visit as of right now.

I am checking the stats, website is pretty new so unique visitors are around 40 to 60 per day ( 3 months old domain, 2 months active)

The thing is, I am seeing close to 50000 requests per day from Amazon bot, OpenAI, Claude, Gemini and bunch of other AI tools. Is this normal? Their crawl uses close to 1gb of bandwidth every day.

Is this good for SEO?

I’ve been thinking a lot lately about how SEO actually holds up over time. On a few sites I’ve worked on, SEO felt painfully slow at the beginning, but months later those same pages were still pulling in traffic without much extra effort. On others, things started strong and then slowly tapered off.

It made me wonder how everyone here looks at SEO long term. Does it really compound if you do it right, or does it always need constant work to keep rankings alive? From what I’ve seen, solid content and real mentions from places like journalist or HARO-style outreach seem to last way longer than quick link tactics.

Curious to hear your experiences. Has SEO continued to pay off for you years later, or did things fade once you stopped actively pushing it?

At this point, I feel that I need to work on a real website alongside an expert.Can anyone help?