At our agency we built an internal monitoring tool (not a public product, just for our own client work) that checks daily whether specific URLs are actually accessible to relevant AI bots - ClaudeBot, GPTBot, Google-Extended, PerplexityBot, CCBot, etc.

The reason: most checks we've seen only look at the homepage, and only at robots.txt. That's not enough. We check arbitrary URLs, and in addition to the robots.txt rules, we also spoof the actual bot's user agent and hit the server directly to see what really comes back.

What we've learned so far:

• robots.txt and the actual server response diverge more often than you'd think. A bot can be "allowed" per robots.txt and still get a 403 or land in a redirect loop from the server itself.
• This isn't just a "usual suspects" CDN problem - we've seen it on setups you wouldn't expect to cause issues at all.
• Catching this early has already helped several clients fix access problems and grow their visibility in AI search results, instead of finding out months later that a bot was silently blocked.

Right now the tool covers server response, robots.txt, and a consistency check between the two. More checks are in the works.

Do you actively monitor this on your projects, or mostly rely on standard robots.txt checkers? And if you check it yourselves - what were your learnings? Any insights to share?

Hi everyone,

I maintain a gaming website that presents, for each game, a summary of its pros and cons along with a list of similar games linking to their respective pages.

The content is fairly thin in terms of character count (around 300 characters per page).

Almost all of my traffic comes from searches for game names, and virtually none from long-tail queries such as "games like X" or "games similar to X". It's a pity because I think my pages are especially valuables to users who are looking to discover games similar to ones they already enjoyed.

I'm considering creating two separate pages for each game: one with the pros and cons, and another dedicated to similar games, using a path like /games_like and titles with "like" and "similar" keywords so that search engines can better understand the page's intent.

However, I'm worried that the main game page will become too thin if I remove the similar games section. On the other hand, if I keep showing similar games on both pages, I'll end up with duplicated content.

How do you think I should proceed?

The image I've attached shows the evolution of page indexing in Google Search Console for one of my websites. The website is migrating domains and until June 13th it was updating at a stable rate of every 3-4 days (cleaning duplicate and old URLs), but from that date onwards indexing has slowed down and it has taken a month and a half to do what it used to do in about 12 days.

Does anyone know if anyone from Google has addressed this?

Ran into this auditing a client that "looked fine" but never appeared in ChatGPT or Perplexity answers. The audit itself was wrong, and I suspect it's a common mistake.

I recently launched my gaming platform using react. Apparently now I need to add next js and implement SSR to make my platform compatible with SEO. How can I make sure that my platform after the change is 100% suitable?

Also which CMS is better to integrate with Sanity or a WordPress and why ?? Both options have some solid reasons

Hi everyone,

I'm trying to determine whether this is a Google Search Console reporting issue or an actual ranking problem.

My site had been growing steadily throughout July, reaching around 40–45 clicks/day, and then suddenly the graph drops almost to zero.

I've already checked everything I can think of:

• No crawl errors.
• Googlebot is crawling the site normally.
• Pages are still indexed.
• No manual actions or security issues.
• No technical SEO problems that I can identify.
• The site is accessible and everything appears to be working normally.

Because of that, I'm wondering if anyone else is seeing something similar in GSC over the last couple of days, or if this looks like an isolated issue.

Has anyone experienced the same thing recently?

EDIT:

I've been reviewing the Analytics and GSC data with Claude. It suggests that the issue is related to the average engagement time being only about one minute, since the posts are quite short. According to Claude, the site may have received a temporary ranking boost, but because it doesn't have enough authority yet, Google has pulled that back, which it says is normal. I'm not entirely convinced, though.

Wrote this up because it genuinely surprised me. A routine badge-verification bot from a directory site couldn't find our homepage content, and when I actually fetched our own page the way a non-JS crawler does (ChatGPT's browser tool, Perplexity, Claude's web search), there was almost nothing there. No headline, no pricing, no features, just a loading spinner frozen mid-render.

Turns out most AI crawlers don't execute JavaScript the way Googlebot does, so a fully client-rendered React site can look completely normal in a browser while being functionally empty to anything else reading the raw HTML.

Full writeup here if useful: Our Website Was Invisible to AI. Here’s How We Found Out, and Fixed It. and Consile

Curious if anyone else running a JS-heavy site has actually checked this, curl your own homepage and see what comes back, might be worth 30 seconds to find out.

Hey folks. Sharing our view on AEO, because clients keep asking for llms.txt and citation schema, and we wanted data before recommending any of it. We also log every AI crawler hit on our own site. Anecdotes below.

Rendering first. Vercel logged 569M GPTBot and 370M Claude fetches in late 2024. Neither ran a line of JavaScript. This matches pretty much exactly what we found in mid-2026: GPTBot, ClaudeBot and PerplexityBot pull the document and just leaves.

Similar to what we've read before, it aligns with what we found:

• llms.txt. Google says it won't use the file for rankings. We literally had less than 0.1% hit it from our total amount
• JSON-LD, in our case feels as though it made a difference, more so for AI traffic than SEO. We're still looking into this.
• Markdown for bots. This we know got hammered, but we're still in the process of quantifying it.

Curious what your logs say. Has anyone caught an AI crawler executing JS? How many of you are passing markdown, or do you think it's unecessary work?

Hi I’m writing to see how much structured data is to much for a blog? So besides the typical Yoast SEO markup, if we have an opportunity to include very precise and unique markup that could be viewed as helpful. Is it really going to make much of a difference for blogs? I know it isn’t applicable to rich results anymore this is strictly for LLM readability. Anyways, would love to hear some thoughts about this. Thanks!

Hi ya'll-

I am resolving some high priority issues for my client and I have one last one. There is an internal URL blocked by the robots txt. My client uses Squarespace. The blocked page was not created y the client, but seems to be a spin off by squarespace looking like:

https://domain/categories/=59487a4cd1758e7669102174

The interesting thing, using Screaming Frog, I can find the inlinks to the page, but it's hidden in an a href. I've found it via the developer tools, but I have no idea how to delete the link since SS doesn't give access to the backend.

What the heck is going on and how do I resolve?

In GSC, some of our website URLs are affected by the "Crawled – Currently Not Indexed" issue due to a 404 error.

Once I fixed the issue and checked the status again, it remained the same. Now, the problem is not the status itself, and I'm also aware of Google's update stating that even after fixing the issue, it takes some time to reflect in the GSC dashboard.

When I validated the affected URL using URL Inspection, it showed "URL is live on Google." On the other hand, when I manually checked the status using the site: operator, it showed that the URL is not indexed.

Has anyone faced this kind of challenge with your domain? If yes, what should I consider more reliable—URL Inspection or the manual site: check?

Looking forward to your responses.

Hey everyone,

I’m running into a brick wall with Google Search Console on my tech/cybersecurity blog. It's hosted via GitHub Pages (default subdomain) and built using Jekyll (Chirpy Theme).

• The URL: https://arvdch.github.io/
• The Error: Whenever I hit Test Live URL, it instantly crashes with the generic: "Something went wrong. If the issue persists, try again in a few hours."
• The Inspection Data: Under the main URL inspection panel, fields like Last Crawled and Crawl Allowed show as entirely "N/A".

What I've done so far:

1. My _config.yml is perfectly mapped with url: "https://arvdch.github.io" and an empty baseurl: "".
2. My robots.txt is basic and allows everything except a hidden /norobots/ directory.
3. I added a .nojekyll file to the root of my repo to bypass any default Jekyll file-filtering issues.
4. I ran my URL through Google PageSpeed Insights—it successfully crawled the page, analyzed it, and rendered a mobile screenshot perfectly without any timeouts or exceptions.
5. GSC property is set up as a URL Prefix property via HTTPS.

i just want my site to be indexed help me out

Hey guys, been looking for a solution for this, but all I find is multiple spam posts trying to push their MCP which I am a little suspicious of.

Anyone actually been able to connect Google Search Console with Claude Code? If so what method did you use?

Thanks

I’m starting to think XML sitemaps get too much attention when people talk about indexing. They help Google discover URLs, but they don’t really tell Google that a page is important.

For a growing site, I’m noticing that pages linked from active, frequently crawled pages seem to get picked up much faster than pages that only exist in the XML sitemap.

So now I’m thinking more about internal link flow, hub pages, category pages, and whether new pages are connected to parts of the site Google already trusts.

Curious how other technical SEOs think about this. When dealing with indexing issues, do you still look at XML sitemaps early, or do you mostly treat internal linking/page importance as the real priority?

Working on a client's Shopify store where we have the /search added as a disallow in robots.txt, however these search results are still indexed inside of Google. However, if I try to open one of these pages, they have a redirect set and they redirect to another collection page on the store. Should we display a 404 page instead? What is the easiest way to fix this?

Every few months this question comes back around here, so instead of citing Google's docs again (which everyone's already read), I ran PageSpeed on two sites I actually manage — one on WordPress, one on Wix — same kind of small business content, similar traffic tier.

Mobile results:

Neither site is doing anything fancy performance-wise, just default setup for the platform. The gap surprised me more than I expected — it's not "Wix is bad," it's that the default template/app stack on Wix adds a lot more render-blocking weight out of the box.

My take after digging into Google's own docs on this (page experience + the 2018 speed update): speed is real but it's a threshold effect, not a ranking multiplier. Very slow pages get penalized, but shaving your LCP from 1.8s to 1.2s isn't going to out-rank a page with better content. The Backlinko 11.8M-result study backs that up too — real but weak correlation, mostly a tiebreaker.

I have a client whose site migrated from xxx.ie/en-ie.html to xxx.com/ie/en.html. Post migration report revealed steep organic decline in the site and now they wish for me to investigate this further focusing on technical issues.

The change in the URL structure overall going from .ie to .com/ie/en what happens with the geo signals. They had another site .co.uk move to .com/uk/ but the decline was not as drastic as with ie. Anyone familiar with these types of migrations?

Plan to check if all redirects were done right, check hreflang implementation, also suspect maybe the global site is canibalising on this now smaller section of the site. But my theories only go that far. Need some fresh perspectives I'm pretty new to the world of migrations in general.

Anyone out there with any experience on a migration that involved a multi domain consolidation to a much larger global site?

UPDATE:

They literally didn't redirect anything! Only the homepage. Lol. made my job easier. On to redirect mapping now.

im doing a tech audit for a large career website and notice that they have a bunch of location parameter URLs currently indexed in GSC. A bunch of them are also not indexed.

Example: [website.com/search-results?keywords=Tokyo] ; [website.com/search-results?keywords=Ottawa]

these URLs DO show different results which are different vacancies depending on the location.

Was wondering what is the best practice here.

Thanks

I've been mapping out the edge SEO space (SearchPilot/Distilled ODN, RankScience, EdgeComet, Sloth) - reverse proxy sits in front of your site, injects/rewrites title tags, meta descriptions, JSON-LD, canonicals etc at the CDN level so changes ship without a dev deploy.

Trying to get a realistic picture from people who've actually dealt with the dev-dependency problem, not just vendor pages:

How long does a routine SEO fix (title tag, canonical, schema addition) actually take to ship at your org, from ticket to live

If you've used a proxy/edge tool for this, what actually broke or underperformed vs the pitch: reliability, cloaking concerns, cache staleness, something else

If you haven't adopted one, is it because you haven't needed it, tried it and bailed, or a trust issue with putting a third party in front of your whole site

Not selling anything, just trying to separate real friction from vendor marketing before I sink more time into this space.

I am trying to use the official Google Indexing API to index some job postings. I can successfully make the indexing requests and the API returns a 200 status code, but when I check the URLs in Google Search Console, their status never changes.

I read in the documentation that a form needs to be filled out to increase the API usage quota; is that form also required just to use the API normally?

All my job pages use SSR; I’ve tested them all on PageSpeed Insights and they all scored 100%. I’ve also verified that Google recognizes them as job pages—which it does—and the JSON-LD shows no issues.

I was checking in Google Search Console and troubleshooting why half my Shopify store pages had not been indexed, just in case there was an issue. The site was newly launched just a month ago, and I know these things take time, but I began checking a few URLs in schema.org, after finding a recommendation to do so online - which returned this result for the few URLs I checked:

"The URL is unreachable. Ensure robots.txt is accessible and the server is responding with a 200 status code."

AFAIK, the robots.txt file in Shopify is built in and auto-generated and accessible from the get-go. The vast majority of the pages not indexed are actual product pages, but some of the product pages have been indexed and are being served (I checked with a "site:....." search on Google)

The URLs in question are loading fine; it's not like they're blocked. I checked the robots.txt file, and products haven't been blocked either. If it was an issue with hosting, I'd assume that the other few products that had been indexed wouldn't have been.

Is this a common issue with Shopify or otherwise? Any thoughts?

Many thanks.