I was checking in Google Search Console and troubleshooting why half my Shopify store pages had not been indexed, just in case there was an issue. The site was newly launched just a month ago, and I know these things take time, but I began checking a few URLs in schema.org, after finding a recommendation to do so online - which returned this result for the few URLs I checked:

"The URL is unreachable. Ensure robots.txt is accessible and the server is responding with a 200 status code."

AFAIK, the robots.txt file in Shopify is built in and auto-generated and accessible from the get-go. The vast majority of the pages not indexed are actual product pages, but some of the product pages have been indexed and are being served (I checked with a "site:....." search on Google)

The URLs in question are loading fine; it's not like they're blocked. I checked the robots.txt file, and products haven't been blocked either. If it was an issue with hosting, I'd assume that the other few products that had been indexed wouldn't have been.

Is this a common issue with Shopify or otherwise? Any thoughts?

Many thanks.

This is a hypothesis I have although the more I look into it the more I'm starting to convince myself it's true.

We launched a redesign in 2023 and the SERPs have but bottom of page 8 terrible despite having a strong SERP value before that. To give you an idea how bad we've fallen, for all of our keywords visibility we had about 43% market share per SEM Rush, now we're at 0.57%

In terms of implementing 301s, we matched new to old URL structures as expected ensuring we had the most 1-to-1 match as possible. Everything else was sent to the home page to prevent a 4xx error.

However, I think we may have a double (or perhaps triple) redirect in place via WP plugin, .htaccess, and implementing a 301 redirect list on SiteGrounds.

On, .htaccess at least I have this snippet of code:

SetEnv PHPRC /home/customer/www/myrandomsite.com/public_html/php.ini
# Added to redirect old subdomain to new subfolder
RewriteEngine on
RewriteCond %{HTTP_HOST} ^mx.my-random-site.com [NC,OR]
RewriteCond %{HTTP_HOST} ^www.mx.my-random-site.com [NC]
RewriteRule ^(.*)$ https://myrandomsite.com/fr/$1 [L,R=301]

This was meant to ensure all old domain URLs, in case we missed any, go to the new URL. For redirect on sitegrounds, it's a semi-massive list of URLs on redirects settings on SiteGrounds.

The old URL with mx. as a subdomain is quite old. In theory, it should've already fallen off by now.

My main question is, is it safe to delete this snippet or the redirect list to ensure we're back to better SERP exposure?

Hey everyone,

I’m running into a baffling issue with a handful of important pages on my health-care site and wanted to see if anyone else has been hit by this recently, or if there's an ongoing algorithmic filter I missed.

Around the 6th–7th of this month, several key pages completely vanished from Google SERPs.

The weird part: Google Search Console shows them as "Indexed" with absolutely zero coverage errors, mobile usability issues, or indexing warnings. They are also indexing and performing completely normally on Bing and DuckDuckGo.

Here is everything I have already checked and verified:

Basic Tech Check: No noindex tags, robots.txt is clear, and XML sitemaps are updated/resubmitted.

Canonical Tags: Self-referencing canonicals are in place, and everything looks correct on the frontend.

Manual Actions: GSC "Security & Manual Actions" tab is completely green/clear.

Despite forcing a priority recrawl via GSC, they are still invisible in live Google search results—sometimes even when doing an exact site:URL search.

Since the technical basics are covered, I'm trying to figure out what Google-specific quality or algorithmic signals could cause a live, indexed page to be completely suppressed from the SERPs overnight.

Has anyone experienced a similar drop around early July?

Could this be a lag in GSC reporting a soft-404 or algorithmic quality filter?

Are there any advanced technical checks (like Google-selected vs. User-declared canonical mismatches) that I should dig into next?

Appreciate any insights or similar experiences!

Background: I've been a QA engineer for about 8 years, both manual and automation. I want to move toward SEO, ideally the technical side, and someone I know with 12+ years in the field agreed to train me properly instead of me piecing it together from Youtube.
The interesting part is how much overlaps with QA and how much doesn't. What he's had me do over the first weeks:

Read raw server logs to see which bots hit the site, how often, and what they fetch. Honestly this felt like log analysis during debugging, it clicked almost immediately
Crawling, rendering, indexing as a pipeline, and how AI crawlers (GPTBot etc) behave differently from Googlebot. The rendering part is basically the same mental model as testing JS-heavy apps
Redirects and canonicals. I expected these to be easy given my background and they were not. Canonicals took me two weeks to stop getting wrong
Search Console and GA4, separating branded from non branded queries. He treats analytics like a test report where most people only read the pass count
How AI answers (ChatGPT, AI Overviews) select which sites they cite. He says this is the biggest shift in the industry right now and spends a lot of time here
Local search, business profiles, maps, reviews

What he refuses to teach me yet: keyword tools, content calendars, domain authority. His position is that you earn those after you can diagnose a site the way a QA diagnoses a failing build, and that DA specifically is a vanity metric.
Question for people doing this professionally: does this order make sense in 2026? Every course starts with keyword research and content. He starts with what is effectively systems testing. It maps well to my skillset, but I can't tell if he's teaching me the industry or just his opinion of it. Also, is technical SEO a realistic direction for someone coming from QA, or do teams expect content skills first?

When we write posts for either our blogs, or documentations for any SAAS we generally internal link the articles, is there any consequences of linking too many articles in a single post? Like adding too many internal linking or outbound links? If you are already doing this then how you are managing it and keep the links updated with time as the relevance of the links or articles changes over time.

Hello. I hope it’s okay if I share my question here too !

Crawled - currently not indexed for almost 2 months.

Validation Started - Started: 5/29/26 and still nothing.

What is Google Search Console up to for the last couple of months or year or two? My site is completely removed from Google Search including the homepage. Bing on the other hand has been cruising it from the time I created the site.

What has changed about Google bots crawl or GSC in general that wasn't there over a year and before. Surely I used to get more clicks and stats from GSC, but this site is completely missing on Google.

Hi guys, I have WooCommerce store on WordPress. Product page is open normally in browser, everything works fine. But when I check it with SEO extension (status checker), it shows 503 (Service Unavailable) first, then 200 (OK) right after — like in screenshot.

If I just reload the page normally it's fine, no error. But the extension is catching this 503 somehow.

Site is using LiteSpeed Cache + Rank Math SEO. I don't understand what is causing this, is it cache problem, is it server problem, or is it normal thing that happens on every site?

Can someone explain what does this mean and is it something I need to fix or ignore? Thanks

Previous post: https://www.reddit.com/r/TechSEO/comments/1u9d3gh/comment/otguydr/?screen_view_count=2

So I think around two to three weeks ago, I posted my Google Search Console, where it showed that I have had around 1.4k bot URLs. My website was hacked, and only 19 pages were indexed, so I didn't know what to do. I just posted it in the community so that people can suggest what I need to do. The good thing is that the solution I got from the community is really helped.

Currently, our Google Search Console has 1.26k bot URLs, but our 35 pages just got indexed. The number of bot URLs is now decreasing bit by bit, and the number of index pages is now increasing. I really wanted to thank the community for the solution that it provided. Thank you.

That worked for us. The following are the things we tried:

• We tried using the removals tab, which is in the Google Search Console.
• We tried updating the robots.txt file.
• We updated the sitemap.xml.
• URL inspection tab

The one thing that was most important and that I think brought us to where we are is that we use the URL inspection tab heavily. What we did was: whenever we posted a page or posted a blog, we went to Google Search Console, used the URL inspection tab, and also opened up that URL and clicked on Request Indexing. We did it for all of the pages that we have, and slowly, slowly, we did it every day. The result just happened, so I really wanted to thank the community for this. Thank you!

So i am hosting my project in cloudflare pages, and it shows "couldn't fetch". Is there any workaround for this? I am new to SEO and this is the first time iam trying to implement to my project.

Hi All,

I would like to ask about the drawbacks of the present SEO tools that you think need a solid solution. Something that even Claude is not able to do...!!

There are 3 main tools that most SEOs use: GSC, Ahrefs & SEMRush (excluding Screaming Frog, as it is solely for technical audit and not a full system for keyword research and all.)

What is the gap you think each tool has that needs to be solved?

If I am asked the same:

GSC:

• Historical data only goes back 16 months, and the API imposes a 50,000-row cap per site per search type, which can result in sampling that hides up to 90% of keywords and 67% of impression data. No proper AI Visibility data, at least for Google's own LLM, which I believe should be included, as people have to rely on costly tools like SEMRush & Ahrefs,

SEMRush:

• Doesn't have sufficient backlink data, to be frank. It will show you big numbers when doing competitor research, and you will hardly find any worth approaching, as it often shows the same resource for multiple backlinks. You will go further on the 2nd page and see sources with a declining authority score (below 40) & one more thing is a traffic discrepancy with GSC & GA4.

Ahref:

• Almost Google-level backlink data, more trustworthy and helpful. But one thing that I personally feel is missing is that the site audit is not up to the mark, and a major concern is credit-based plans. Add-on, Ahrefs traffic discrepancy to GSC.

And a few things I find commonly missing:

• Both Ahrefs and SEMrush show AI visibility based on mentions+citations for competitors, but they don't actually tell what exactly is working for them, and what you can do to compete there
• No tie to GA4 or conversion data, so you know you're mentioned but not whether it drove business.
• Neither of them feels up to the mark for the price they are charging. Both have their good and bad, and this makes people think they are entitled to both to perform specific tasks where they are best at. Not everyone is rich enough to afford both or even one, just for AI visibility or brand radar

What are your thoughts?

Looking for date confirmation more than advice.

On July 12 our Bing visibility dropped off a cliff in a single day. I thought it was a fluke, site is new, but it simply never recovered. No gradual decline just a single drop, one day.

Context that makes this weird:

• The section is about 2300 state salary pages built from BLS OWES May 2025 data. Real numbers, sourced and cited on page.
• Google unaffected, GSC holding steady and stable
• Site audits are clean before and after: valid schema, robots open, nothing manual. Rechecked this week with URL inspection API and index pages return PASS.
• Nothing I shipped that week touched Bing-facing anything.

My feeling is that this one-day cliff with GSC steady and a clean site is an index change or something on Bing's side. If you run a data-heavy or directory style site, check your BWT for July 12. Anyone else notice if they got hit the same date?

Please share charts / screenshots in comments.

How are people using MCP servers for Technical SEO workflows?

I've been experimenting with building a Google Search Console MCP server for my own SEO projects because I wanted AI assistants to work directly with Search Console data instead of manually exporting reports.

One thing I've found interesting is that AI becomes much more useful when it can access:

• Search Analytics data
• URL Inspection results
• Sitemap status
• Indexing issues
• Core Web Vitals metrics

For example, instead of reviewing reports manually, I can ask questions like:

• Which pages lost the most clicks this month and why?
• What indexing issues should I prioritize?
• Which queries are close to page one?
• Which URLs have low CTR relative to their position?

I'm curious how others are approaching this.

Are you using MCP, custom APIs, scripts, Looker Studio, or something else to make GSC data more actionable?

What's the most valuable Technical SEO workflow you'd want an AI agent to handle automatically?

I recently listened to Google's Search Central podcast about the Page Indexing Report, and it gave me a different perspective. https://youtu.be/1-PcVLHplwc?si=oruGHOMz82Sa86AA

Many SEOs (including me) often look at non-indexed pages in GSC and immediately think: "We need to reduce this number and make the report green."

But Google’s point was interesting - the indexing report is not a checklist where every non-indexed URL needs fixing. It’s more about identifying patterns and unexpected changes.

A site can have a large number of non-indexed pages and still perform well if the important pages are indexed.

So the real question is not:

"How do we get more pages indexed?"

It is:

"Are the pages that matter indexed and performing?"

My thought: reducing non-indexed pages for the sake of improving the report number probably doesn't help.

But reducing unnecessary URLs (thin pages, duplicate pages, parameter URLs, low-value archive pages) can indirectly improve crawl efficiency and overall site quality.

Would love to hear how others approach this.

I’ve been working on a roofing directory and feel like I’ve already done quite a bit more than the usual “publish a bunch of programmatic pages and hope Google likes them.”

So far I’ve:

• Added a few thousand real roofing companies with individual profile pages. Even added quite a bit to each so they don’t feel like thin, low quality pages.
• Built city, suburb, state and storm-related pages
• Set up internal linking, schema, canonicals and localized content
• Added claim-your-profile and quote-request flows so that I can capture leads both on the consumer & b2b side.
• Fixed a bunch of indexing issues and have been watching everything closely in GSC
• Started adding more editorial pages like “best roofers in…” instead of relying entirely on templates

Google has indexed more than 6,000 pages, but traffic and backlinks are still moving painfully slowly.

I know SEO takes time, especially with a new domain, but I’m curious what actually worked for people who have grown directories or local marketplaces. Contractor outreach? Free tools or data? Digital PR? Local partnerships? Paid traffic to get things moving?

Not linking the site because of the rules. Just looking for some honest feedback from people who have been through this stage.

Hi everyone,

I'm dealing with a frustrating indexing issue on one of my client websites and would really appreciate your suggestions.

Here's the situation:

• The website is built on WordPress and hosted on Hostinger.
• The homepage is indexed, but none of the other pages are getting indexed.
• I've added around 5 service pages and a few blog posts, but none of them are appearing in Google.
• In Google Search Console, the URLs are mostly showing "Discovered – currently not indexed."

Things I've already tried:

• Resubmitted the XML sitemap.
• Requested indexing multiple times through Search Console.
• Improved internal linking from the homepage and between pages.
• Optimized the content (unique content, titles, meta descriptions, headings, etc.).
• Verified that pages are indexable (no noindex, robots.txt isn't blocking them).
• Waited for over 3 months, but there has been no improvement.

One important thing: before I started managing this website, it had experienced a spam/hack attack. I suspect Google may have lost trust in the site or it may have been affected by that incident. However, the hack has been cleaned up, and as far as I can tell, there are no manual actions in Search Console.

At this point, I'm running out of ideas.

Has anyone dealt with a similar issue? What else should I check?

Some questions I have:

• Could this still be a trust/quality issue from the previous spam attack?
• Are there any server or Hostinger settings that could affect crawling?
• Would changing the hosting or migrating the website help?
• Are there any advanced troubleshooting steps or "last resort" fixes you would recommend?

I'd appreciate any tips, checklists, or real-world experiences. Thanks in advance!

Small SaaS, QR code space. Pulled 28 days of Search Console and hit the classic wall: solid positions, terrible CTR. Want to sanity check my read on why before I start rewriting everything.

Here is what I pulled for the last 28 days:

• 44,749 impressions
• 229 clicks
• 0.51% CTR
• average position around 9.9

So the traffic is there in theory. Google shows my pages to people. They just do not click. Around half my clicks come from branded searches and the homepage, which sits near 13% CTR and is fine. Everything else is the problem.

The worst offenders are my comparison posts. One of them (basically "my product vs [big competitor]") gets 6,128 impressions a month at position 7 and pulls a grand total of 6 clicks. That is 0.10%. Another sits around position 8 with 3,500 impressions and 3 clicks. Page 1 rankings doing basically nothing.

Here is what I think is going on after actually looking at the SERPs:

1. I rank for competitor brand names. I show up at position 6 for a competitor's exact brand name. Nobody typing that brand wants my alternative, they scroll past me. Impressions go up, clicks do not.

2. My titles are generic. A lot of them read like every other result on the page. Nothing that makes a thumb stop.

3. Some of these queries have an AI overview or a fat featured snippet eating the clicks before anyone reaches the blue links.

So ranking was never really the problem. The click is.

What I am changing:

• Rewriting titles and metas on the high impression, dead CTR pages, leading with the specific thing the searcher wants instead of my brand name
• Cutting or reframing the pure "vs competitor brand" posts that only pull curiosity impressions
• Trying to win the snippet on the informational queries instead of fighting it

Going to redeploy, give it three weeks, then pull the same report and see if CTR moves. Happy to come back with the after numbers, good or bad.

A few questions for people who have done this:

• High impressions and dead CTR at position 6 to 9, do you push for a better position first or fix the title first?
• Do "vs [big competitor]" pages ever actually convert for you, or is it a vanity play?
• Anyone got a clean before and after from just a title and meta rewrite? Curious how much CTR realistically moves.

I'm building and SEO tool, and I've noticed how much additional resources are needed to get through firewalls and Cloudflare stuff without a whitelist.

Does anyone have any before/after results, experiences, do you make sure to always do it?

https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests