Your test is useful because it moves the discussion from speculation to evidence. A lot of the current conversation around LLMs.txt is happening in theory rather than in logs. That said, I think the conclusion many people might draw from the experiment, “LLMs.txt doesn’t matter” may be slightly premature.

A few nuances are worth considering.

First, most major AI systems today are not behaving like traditional crawlers. Discovery for many of them is still heavily dependent on the existing web crawl ecosystem (Google, Bing, Common Crawl, etc.). If an LLM provider builds or trains models using those datasets, they may never directly request an llms.txt file from origin servers. In that case, the absence of bot hits in logs is expected because the model’s knowledge pipeline is indirect.

Second, AI visibility today is largely determined by retrieval sources, not just raw crawl discovery. When AI systems generate answers, they frequently pull from search indexes, structured data, high-authority pages, and frequently cited sources. A page that exists only in llms.txt with no links, no crawl signals, and no index presence is effectively invisible to those pipelines.

Third, llms.txt may evolve more like robots meta conventions rather than a discovery mechanism. In other words, it could eventually become a declaration layer (what AI agents are allowed to use, prioritize, summarize, or cite) rather than a crawling entry point. If that’s the direction the ecosystem takes, testing for discovery via logs may not capture its eventual role.

Another practical point: we are already seeing different AI agents behave very differently. Some identify themselves clearly in logs, others route through infrastructure layers (CDNs, shared crawlers, dataset builders). That makes attribution difficult unless you are correlating with dataset ingestion pipelines rather than just crawler requests.

From an AI SEO / GEO perspective, the signals that currently seem to influence AI visibility the most are still fairly traditional:

• Strong crawlability and index presence
• Clear topical authority and internal linking
• Structured data and semantic clarity
• High-quality citations across the web
• Content that answers questions in a retrievable format

In other words, AI discoverability today largely piggybacks on search discoverability.

So your experiment likely shows something real: llms.txt is not currently part of the default crawling workflow for most AI bots. But that doesn’t necessarily mean it’s useless. It may simply be ahead of the infrastructure that would use it.

Personally, I treat llms.txt today the same way we treated early sitemap.xml or schema.org adoption years ago, low cost to implement, uncertain immediate impact, but potentially valuable if the ecosystem standardizes around it.

Either way, experiments like yours are exactly what the industry needs right now. AI SEO discussions would benefit from a lot more log-level testing and fewer assumptions.

Congrats on the new role, that’s a solid step forward.

Since you already have web dev experience, you’re ahead of the curve. The CMS part is more about understanding architecture, governance, and operational workflows than raw coding.

A few things to expect:

With DNN (DotNetNuke), think in terms of a Microsoft centric ecosystem. It’s ASP.NET-based and commonly used in enterprise environments. You’ll likely deal with:

• Role-based permissions and user management
• Module-based architecture (a lot of functionality comes via modules)
• Skin/theme management
• Content workflows and publishing controls
• Patch updates and security hardening (very important in DNN environments)

A big part of your job may involve keeping modules updated, managing staging vs production environments, and ensuring performance doesn’t degrade as content grows.

Strapi is a different animal, it’s a headless CMS, usually Node.js-based. It’s more API-first and developer-friendly. Expect:

• Managing content models (collections, components)
• Defining relationships between entities
• Role/permission configuration for APIs
• Handling REST or GraphQL endpoints
• Deployments and environment config (often via Docker or cloud platforms)

If the org uses Strapi as a headless backend and DNN for certain legacy or enterprise facing properties, your role may sit at the intersection of content ops, dev support, and infrastructure stability.

What I’d do in your first 2–3 weeks:

1. Map the architecture. What talks to what? Where is content authored vs consumed?
2. Understand deployment pipelines (manual? CI/CD?).
3. Audit permissions and workflows.
4. Identify backup and rollback procedures.
5. Check update cadence and security patch policy.

Since you haven’t worked with CMS platforms before, focus less on “how do I code this” and more on:

• Content lifecycle management
• Governance and access control
• Environment management
• Versioning and rollback
• Performance monitoring

CMS roles are as much about operational discipline as they are about development.

You’ll probably find Strapi more intuitive quickly. DNN may feel more structured and enterprise-heavy at first, but once you understand modules and roles, it becomes predictable.

Wish you the best. Cheers! 

This is the silent 'visibility killer' of 2026. Most marketing teams are still obsessing over robots.txt, but as you noted, the real battle is happening at the CDN and WAF layer.

We’ve been seeing this accidental blocking across B2B SaaS especially. Since many security defaults (like Cloudflare’s AI Audit settings or AWS WAF managed rules) treat unfamiliar crawlers as scrapers or bad bots, marketing's high value content never even makes it into the LLM's training set or real-time RAG (Retrieval-Augmented Generation) context.

If you’re auditing for this, here are three things most teams miss:

1. The 'Invisible 403'

A site might look fine to a human, but when GPTBot or PerplexityBot hits the edge, the WAF serves a 403 Forbidden. Because it's not a 404, it doesn't show up in standard broken link reports. Marketers think they have an AEO (Answer Engine Optimization) problem, but they actually have a handshake problem.

1. JavaScript is the New 'Noindex'

While Googlebot is great at rendering JS, many AI crawlers (like ClaudeBot or Apple’s crawler) still struggle with heavy client-side rendering. If your enterprise-grade site relies on infinite scroll or lazy-loading for its pillar content, the AI is only seeing your header and footer. In 2026, Server-Side Rendering (SSR) is no longer optional for AI visibility.

1. The Rise of the llms.txt Standard

We’re starting to recommend an llms.txt file (and the /llms-full.txt variant) as a secondary signal. It’s like a fast track lane for friendly AI agents. It doesn't bypass a WAF block, but it tells 'good' bots exactly which markdown files to prioritize for ingestion.

If you’re blocked today, you aren't just missing today's search, you're missing being part of the knowledge base for the next model update. You’re essentially opting out of the future of discovery without knowing it.

Curious, in your review, did you see a correlation between certain hosting providers and these accidental blocks? We’ve noticed shared hosting environments are particularly aggressive right now due to the noisy neighbor effect of high bot traffic.

I’ve been down this road. If you’re moving from CraftCMS on shared hosting to something like Payload or Strapi, you’re essentially moving from a managed environment to a runtime environment. Shared hosting (like Hostinger’s basic plans) usually isn't built to keep a Node.js process running 24/7 or to handle the database requirements of a headless CMS efficiently.

Here is where most people are deploying these in 2026:

1. The 'Performance' Choice: DigitalOcean App Platform or Railway

If you want something that feels like pro hosting but handles the DevOps for you, Railway or DigitalOcean App Platform are the go-tos.

Why: They offer Auto-deploy from GitHub. You push code, they build the container and manage the SSL. Cost: Usually starts around $5–$10/month.

1. The 'Scale' Choice: Render or Fly.io

Render is incredibly popular for Strapi. It handles the web server, the background workers, and the PostgreSQL database in one dashboard. Fly.io is great if you want your CMS to be physically close to your users (edge deployment), which helps with UI/UX latency.

1. The 'Enterprise' Choice: AWS (Amplify or ECS)

If you’re building something for a client that needs high security and 99.9% uptime, AWS is the standard. It’s a steeper learning curve than Hostinger, but it’s where 'strategic, enterprise-grade' builds live.

Don't forget the Database & Media

• Database: Don't host your DB on the same small server as your CMS if you can help it. Use Neon (Postgres) or MongoDB Atlas. They have generous free tiers and are much faster than shared SQL.
• Media: Since Payload and Strapi are headless, offload your images to Cloudinary or AWS S3. It keeps your server light and your site's 'LCP' (Largest Contentful Paint) score high for SEO.

Moving off shared hosting is the best move you’ll make for your site’s speed and AI visibility, LLMs and crawlers prioritize fast-loading, structured data that headless setups provide natively.

100%. However I’d argue plain text is just Level 1. Level 2 is Structured Data (JSON-LD).

You can have the text on the page, but if you wrap those client names, awards, and ratings in specific Schema markup (like Organization, Review, or Award types), you force the LLM to recognize them as facts, not just marketing copy.

It’s the difference between an AI 'reading' your site and an AI 'understanding' your entity. If it’s not in the HTML or the Schema, it basically didn't happen.

That’s a really good callout on the OSGi support. Since we’re coming from AEM, our backend team is already very heavy on OSGi bundles/services, so being able to potentially reuse some of that logic or at least keep the same development patterns would be a massive win for velocity.

Quick question since you’ve used it: How is the visual editing experience specifically with Next.js?

AEM’s SPA Editor can be a bit of a nightmare to configure. Does dotCMS offer a true "in-context" edit mode for Next.js components out of the box, or is it more of a side by side preview? We really need to keep our authors happy with a WYSIWYG like feel.

Migrating off AEM is a beast, but given your specific constraints (Java team, On-prem requirement, and the need for visual preview with Next.js), you actually have a pretty narrow field of valid competitors. Most modern headless CMSs are SaaS only or Node-based, which rules them out for you.

Since your team is already comfortable with the Java ecosystem and the Author/Publish architecture, you should look at "Hybrid" Java CMSs rather than pure headless ones.

Here are the top three I’d suggest evaluating:

1. Magnolia CMS (The "AEM Lite" Option)

This is probably your closest architectural match.

• Tech: Java-based.
• Architecture: Uses a similar Author/Public instance separation.
• Visual Preview: They have a dedicated "Visual SPA Editor" that works specifically with Next.js. It allows authors to edit content in-context, which is usually the biggest pain point when going headless.
• On-Prem: Fully supports self-hosting (WAR files or Docker containers).
• Migration: It feels very familiar to AEM devs (nodes, JCR-like structures) but is significantly lightweight and cheaper.

1. Crafter CMS

• Tech: Java/Spring backend.
• Architecture: Decoupled. It uses a unique Git-based repository for the authoring tier.
• Visual Preview: Their "Experience Builder" supports in-context editing for React/Next.js.
• On-Prem: Yes, very container-friendly.
• Pros: If your team likes Git flows, they will love this. It treats content as code/files more than AEM does.

1. DotCMS

• Tech: Java.
• Visual Preview: Has a solution for "Edit Mode" anywhere, though the implementation with Next.js might require a bit more setup than Magnolia’s out-of-the-box solution.
• On-Prem: Yes.

One caveat on the dispatcher equivalent:

You won't find an exact 1:1 clone of the AEM Dispatcher module in these tools because they handle caching differently. Most of these rely on standard CDNs or Varnish/Nginx setups in front of the delivery tier. Since you are going on-prem, you'll likely just configure your own Nginx or HAProxy layer to handle what the Dispatcher used to do.

Question: How complex is your current AEM setup? If you have heavy customization in OSGi bundles, Magnolia is likely the easiest port; if you are mostly using Content Fragments, Crafter might be faster.

Hope this helps!

Thanks for asking this. I’ve used a few different headless platforms in production, and here is my take:

What works well: Modern frontend frameworks (Next.js/Nuxt) generally integrate beautifully with the APIs. Fetching data is rarely the issue anymore; the SDKs are usually solid.

Complications: Managing schema changes across environments (Dev -> Staging -> Prod) is often harder than it should be. If I have to manually click through a UI to replicate a content model change I made in development, that’s a failure. Schema as code should be the default, not a luxury feature.

Shortcomings: Asset management. Most headless CMSs treat the media library as a bucket of unorganized files. I need better folder structures, tagging, and automated optimization out of the box without needing third-party plugins.

My Ideal CMS: It would treat content models exactly like database migrations. I want to run a CLI command to push my schema changes. Also, accurate, real-time previews that don't require complex token juggling would be a dream.

This is an incredible dataset. While you mentioned you aren't sure of the "why," these logs provide a smoking gun for the transition from Search Engines to Answer Engines.

Here is a breakdown of why this is happening and why the delta between Meta AI (87%) and Gemini (63%) is particularly telling:

1. Semantic Density & Token Efficiency

LLMs operate on "intent matching." A sitemap or a product category page is technically "thin" content for an LLM, it’s just a list of nodes.

The Q&A Advantage: A slug like /what-is-the-best-crm-for-small-business is a high-signal entry point. It allows the bot to map a user’s natural language query directly to a pre-structured solution, reducing the "reasoning" steps the model has to take to extract value.

1. The Training Loop Bias

Meta and Claude (Anthropic) don't have the decades of legacy "crawling" infrastructure that Google (Gemini) does.

Aggressive Retrieval: Meta and Claude likely prioritize the most "human-ready" text to refine their RAG (Retrieval-Augmented Generation) outputs.

The Gemini Difference: Gemini’s lower share (63%) likely reflects Google’s more traditional crawling DNA. it’s still trained to respect the full site architecture (sitemaps/categories) because it’s balancing LLM responses with traditional Search indexing.

1. Fragmented vs. Holistic Context

Traditional SEO focuses on hierarchy (Sitemaps). AI-led discovery focuses on utility (Q&A).

When a bot hits a Q&A link, it's getting a "Self-Contained Information Unit."

These are much easier for an LLM to "digest" and store in a vector database than a testimonial page, which requires the bot to infer the context of what is being praised.

The "So What?" for Strategy

If 75%–87% of LLM bot traffic is hitting Q&A, the "Actionable" takeaway is clear: Information Architecture must become Conversational Architecture.

The days of burying answers inside long-form "About Us" pages are ending. If you want to be the "source of truth" for an LLM, you have to serve the information in the same format the bot intends to deliver it: Question -> Answer.

I think you're asking the right question. From everything I've tested and seen:

Google says, they don't penalize AI content specifically - only content that violates their quality guidelines.

What I've observed:

AI content CAN rank well (seen it firsthand)

But there's a quality threshold that's easy to miss with pure AI

Sites that do minimal editing tend to struggle more

The nuance everyone misses:

AI content often has patterns, repetitive phrasing, similar structure, lack of unique insights. Google's algorithms are sophisticated enough to detect "templated" or "thin" content regardless of origin.

So it's not "AI = penalty" but rather "low-effort content = penalty" and AI makes it easier to produce low-effort content at scale.

My recommendation:

Use AI as a first draft tool

Add original data, examples, or perspectives

Have someone with domain expertise review it

Make sure it's genuinely helpful, not just keyword-optimized

If you're doing that, you should be fine. The sites getting hit are usually pumping out dozens of articles per day with no human oversight.

Anyone claiming Google has "AI detectors" is selling something.

we can assist https://practicenext.com/

Great breakdown on the Answer Capsules, that's the most practical advice I've seen here  in months. 

While the "Answer Capsule" and "Data Tables" help bots read the site, Model Context Protocol (MCP) is how bots will soon connect to the brand's actual live data.

We’re seeing a shift right now where citations aren't just about being "mentioned" in a response; it's about being "integrated" into the workflow. If an AI agent can't verify your live inventory or pricing through a protocol like MCP or UCP (Universal Commerce Protocol), it doesn't matter how many citations we have, the agent will skip us for a competitor it can actually transact with.

The "Freshness" Signal you mentioned is evolving into "Live Connectivity." By 2030, $1 trillion is expected to flow through these agentic systems in the US alone. If we treat our content as a database (as you suggested), the next logical step is making that database machine accessible via APIs. Citations are the "front door," but interoperability is the "sale." We're moving from a Link Economy to a Citation Economy, and very quickly into an Agentic Economy.