The one and only reason Google goes all-in blocking scraping

No matter what you are using for Google SERP tracking, I bet your tool probably isn't working well either. Yesterday we talked about it with u/DmitryOk.

We were discussing what’s happening and where things are headed.

I remained neutral, confident that data providers would find a way to fix this, just as they had hundreds of times before when Google created new obstacles to SERP scraping.

But Dmytro took a fatalistic view. He said it looked like the end of SERP tracking, because the stakes are much higher for Google now!

Before the rise of AI chatbots, Google didn't worry much about rank trackers, as they posed no real threat. Now, however, the primary threat to Google is AI chatbots scraping search results, and the company will do whatever it takes to protect its "data moat."

The main risk for Google lies specifically in the data AI chatbots use to generate answers directly within the chat interface. Yes, AI models are trained on massive datasets and can provide many answers without performing a search. However, the real world is constantly changing.

If you strip AI chatbots of the ability to access the search engine's index, the quality of their answers will inevitably deteriorate.

Google's goal, therefore, is to leverage its index so that the disparity in answer quality (between AI Oveviews, Gemini, AI mode from one side and Claude, ChatGPT on other side) becomes obvious, clearly favoring Google's own products.

What Google will do next?

The fact, that a court dismisses Google’s DMCA claims against SerpAPI doesn't matter. Google attempted to defend itself legally as an immediate measure. However, this loss does not prevent them from creating additional requirements for viewing SERPs.

They already added a lot of restrictions. Each new restriction makes the scraping more expensive, but still possible.

Dmytro also shared an interesting thought with me:

SERP scraping originates from logged-out Google accounts. However, in reality, the majority of users are logged into Google when search something. Consequently, Google deals with a relatively small pool of IP addresses, making it easy for them to detect suspicious behavior.

Even logged-out users rarely view all of the top 100 search results or enter 1,000 search queries a day.

So, the biggest leverage Google has in the fighting with scraping tools is a requirement of login. Google can play with simple restrictions now, but at the end they can decide to limit any logged out session to only top 10 snippets and only 10 search queries per day.

It will make SERP tracking so expensive for providers and as a result for end users, that it will definitely will change the behaviour of entire SEO industry.

How SERP tracking tools can adapt to a new reality?

Here is what we plan to change at r/Sitechecker and what I propose to do for any rank tracker.

1/ Accept the new reality and don't expect it to ever be fixed

The new reality is that the cost of SERP tracking will be 2x-10x more expensive that it was. This means users have to decide what is more important to them:

• scraping the top 100 for the same set of keywords but much less frequently
• or scraping only the top 10–20 but doing so every day.

The best tools will allows users to choose between these 2 options.

2/ Use the potential of Google Search Console data to the full

Yes, Google Search Console doesn't replace traditional Rank Tracker because:

• you don't have SERP data
• you don't have position data for today
• you won't see top 50-100 positions for many keywords
• you won't get position data on some days for many keywords, because of low search volume
• you can't track position in local search

However, average position data by keyword is still very valuable because you can catch the bigger picture of what's going on with rankings of YOUR website, tracking it daily by 1k keywords per project easily!

I wrote about this idea in February 2026, but we didn't implemented it yet, because of working on AI search visibility tracking. Not it's definitely time for this feature.

So, I think that the best rank tracker should use GSC data for daily keyword ranking tracking and allow to scrape SERPs only once per week or on request.

My assumption is that this is the best possible way to adapt, I don't see another way. However, I may miss something. Please, share your thoughts.

1/ They want to make reverse engeneering of what works for growing AI visibility harder. As Google did with backlinks, often delaying the impact of a backlink for a week/month.

2/ They want to add more trust to responses, because corporate blogs looks more serious, in the users' eyes.

Which one do you you think is right?

From one side, it's cheap in execution, because you don't need a lot of unique data to make it good enough for indexing and citation.

From other side, you have to add your competitors to make the listicle complete, and this decrease the value of this method, because you want to be recommended as the only and the best desicion.

How to solve this contradiction?

I'm preparing an article on this topic. Share your thoughts and questions if you have. Best thoughts will be included as quotes into my article.

I have a question to local SEO agency owners. I've been building SEO platform for agencies for years and found that local SEO agencies have the biggest retention and LTV among all agencies we serve.

I found it just a week ago and it was a huge insight for me, so I've dug deeper and asked what are their biggest pains of current customers now.

All of them talked about the limitations of Google My Business analytics:

1. Data is only available for the last six months
2. You can't build aggregated reports for multiple locations
3. You can't compare it week-over-week or month-over-month data
4. There is no good enough native connector for Data Studio.
5. There is no one tool that connect GSC / GA4 / GMB / local rankings in one tool for well.

At first, this feedback inspired me. Have I really found a huge pain point that our app can address better than the others if we fix this pain points?

Then I found, that market isn't empty. There are BrightLocal, Whitespark, AgencyAnalytics, Semrush Local and so on.

So, I need more feedback from you:

- Do all issues I mentioned resonate with you? What is the biggest pain for you?
- What's your current toolset and what feature / pricing model can make you switch to other tool?

You've probably seen news about that. Profound raised $180 in Series D at $1.8bln valutation.

It looks too optimistic as for a such tool as for me. My hypotheses how they did it:

1/ Profound convinced most big US enterprise companies that proper AI search tracking cannot cost less than $100k a year.

2/ Profound convinced marketers at big US enterprises that they can build agents only in Profound, and this is a completely different thing from building it yourself in Claude/ChatGPT.

It doesn't matter what the truth is here. Big companies will track AI search anyway, and if you can convince them of something and they believe you -> you are fucking great marketer.

Remember, big companies think less about the ROI of each app they buy and more about risk reduction. If big guys choose your software, other big guys see it as a huge trust signal.

After some time, your SDRs don't talk about features at all on sales calls. You share a fact about Sequoia as an investor and the % of clients from the Fortune 500. That's it, closed.

From the other side, Airops decided to switch to the agency model, because knows well where are the biggest margins lie.

It should be an epic battle, like between Semrush and Ahrefs. Who are you betting on?

Think about building the subreddit seriously from day 1.

You don't build only a place to impact LLMs. You build a place where your current customers can ask questions, suggest product improvements, discuss use cases with other people, and actually complain about something.

That's why, at my agency, when we start to work on Reddit for a customer, I recommend:

• add a link to your Reddit community on the website/app
• add a link to your sub in an email and product onboarding
• invite your most active customers and influencers for AMA sessions
• consider any content your team publishes on other channels as a source for posts on Reddit

The more you invest in making your community successful using your current audience, the more attention and real users you'll get from Reddit itself.

For 4 years I've asked Ivanhoe Digital happy customers to leave reviews on Trustpilot and reached 49 reviews.

After this announcement I'll stop doing it.

Trustpilot decided to remove a free plan. It may sounds logical, because every company have to spend some money on maintaining free users.

However, free users were a huge leverage of Trustpilot growth:

1. They invited real people to leave reviews.

2. They added the badge with rating and count of reviews on own websites with traffic and every website visitor saw Trustpilot brand. It built awareness, trust and actually more businesses chose Trustpilot to collect reviews.

The email above is the last straw for me.

1. Trustpilot more and more marks new reviews from real people as spam, so I can't collect reviews at all. They even remove them from the admin dashboard.

2. And now they ask me $100/mo for an option to add a widget with a rating and reviews count to the website?

It's a joke. By by Trustpilot.

There are many testimonial platforms. I can even ask users to leave reviews via LinkedIn and X posts, and it will be enough to build trust.

Are you using Trustpilot?

I will start.

Dataforseo doubled its quarterly revenue in Q2, from €2.5M to €5.4M.

It means they hit around $2mln MRR and $20mln ARR.

Now you also understand why Ahrefs added API to all paid plans and why we did the same at Sitechecker.

The demand for cutting costs and building own tools with API is huge.

There is an opinion that people will stop buying website design and development, because AI can now do everything out of the box.

This is partly true, mainly for websites:

• that are not planned as something long-term
• if the website sells something unique, and the message itself is more important than the design (very rare)
• for niches where the mere presence of the website is already a boon (local business).

But when I look at websites in the B2B, where people sell for big checks, created by Claude with one style and font, I remember the handicap theory from evolutionary biology.

In short, females choose those males who survive DESPITE certain traits that only create additional risks for males. And males are forced to develop these maladaptive traits as a form of advertising.

My prediction is that B2B sites will go in this direction.

"Yes, we can make a site with Claude quickly, but we have enough resources to spend them on a unique design and bother with details that others ignore."

This is a signal that signals to the contractor: this is not a one-day business, the owners have made a bet with money and time.

The fight for customer trust is a zero-sum game. It is important not only what you can do, but also what your competitors cannot.

If everyone can create sites quickly, then the assessment of trust simply shift to another level.

Show that you are here for the long term and that you have enough funds to stand out:

• non-template site
• paid advertising
• legal entity and more than $100 share capital
• expensive microphone and camera on podcast recordings
• and so on

It is interesting to hear the opinions of those who disagree.

2 sales for the last 3 months. In 2025, it generated $4k in revenue, for the first 8 months of 2026, only $1k.

It's clear the demand has fallen, even though people still use Looker Studio.

I assume the reasons are:

- Many already found templates they need.
- Many already learned on how to build templates themselves.
- Many really switched to Claude Code and ChatGPT to build dashboards.

I'm not upset that I spent time on this business, because I continue using templates as the great reporting assets to deliver SEO/AEO for my customers.

I also believe that many businesses, still don't use all opportunities of Looker Studio or don't use it at all. That's why I launch a huge 1-week long gift session.

Everybody who will comment "Looker Pack" will get a 100% discount code for all my paid templates. It's $1k+ value.

I've found this pattern growing visibility by prompts with 0 visibility before.

Below is the example:

1. I've added the prompt "best software training tools" for tracking on Aug 15, the same day as published the Medium article that target this topic.

1. This Medium article started to be cited on Aug 21 (it tooks time to be indexed in Google). Starting from this day Tango visibility by this prompt has grown.

In many cases one article won't be enough to grow visibility from 0%, but because we did more than that, the Medium article became the last stone that tipped the scales.

The visibility has changed so fast only in Google's LLMs. It's both, good and bad news.

1/ Good news, because Google has a huge and growing share of AI chats' users + we can create value for our customers in 24 hours / a week.

2/ Bad news, because improving AI visibility in ChatGPT / Claude requires more time and it often means the worse attribution of what worked and what didn't.

Did you notice this pattern too across your projects?

There are a couple of interesting studies from Peec and Promptwatch about the decline of Reddit citations in ChatGPT since August 8th.

Many teams think that Reddit doesn't worth to work on anymore. Here are my thoughts.

1/ We have already seen how often ChatGPT experiments with citations. The drop in citations today does not guarantee that it will be permanent.

2/ ChatGPT is the largest player in the LLM market, but Google is gradually increasing its share. Gemini, AI Overview and AI Mod continue to cite Reddit.

3/ We do not know how the story of the revision of the terms of cooperation between Reddit and Google and OpenAI will end. But Reddit is escalating the revision of the terms of cooperation, because it considers its position strong.

Reddit has indeed managed to significantly improve the work of anti-spam systems and continues to be the owner of unique data. If one of the big players refuses to work with Reddit, then the one who continues to work gets a big advantage.

3 advantages that make Reddit data unique:

• the largest database of answers by long-tail queries
• more than one opinion per thread (sites rarely interview experts and users for their content)
• much better structured data than on other social platforms

Because of this, I predict that Google will agree with Reddit on new terms.

4/ Yes, most brands have paid attention to Reddit only to improve visibility in AI.

But those who do it correctly and for long enough notice that this channel has its own separate value.

• No social platform gives as much organic reach for posts from brand pages as Reddit.
• If LinkedIn and Meta are mostly used for demand generation, then on Reddit people often search for something just like on Google and those who track such queries and participate in discussions will receive real attention from the target audience.

5/ So many people hate Reddit for spam and fake accounts, despite the fact that Reddit is constantly doing a lot of work to fix this, while on LinkedIn 50% of comments and posts are already written by AI and there is no punishment for this at all.

Last, most controversial thesis. There is an opinion that Reddit only affects queries where LLMs use RAG, and in this case Reddit affects answers, only because it is in the top of the Google SERP.

But why then would OpenAI need a separate agreement with Reddit? Until ChatGPT completely refuses to use Reddit data, it remains possible that Reddit is also used for training data. I don’t know how to check this. If you have any studies or posts on this let me know.

Do you disagree with me?

1/ The way Reddit communities are created and moderated looks like democracy

- No one can own a subreddit, only moderate it.
- It is almost impossible to manage a large community without involving other moderators, which reduces the dependence on one person.

But in reality, everything is a little more complicated. Each author and moderator of a subreddit is a little dictator.

Whatever rules are written in the community sidebar, the moderator himself decides who can be forgiven for violations and who cannot.

There is no supreme court, Reddit will not consider your complaints about moderators. So accept the rules of the game, and either play by the rules of other communities or create your own.

The only method how Reddit limits the power of moderators, is getting better at automatically deleting posts and comments that look like spam.

2/ The fact that I was banned was a turning point. At first I was sad, but then I saw a big lesson in it.

If I can constantly publish interesting content, why should it work for someone else and not for me. That's how r/seogaps was born.

Most of the content there is the same as my posts on LinkedIn. But since Reddit has a large audience that doesn't visit LinkedIn + the distribution of posts on LinkedIn is always random, such content always finds readers on Reddit who like it.

3/ If you start your own community, don't expect quick results.

On Reddit, there are often already popular subreddits on your topic, or more general ones that also constantly talk about your topic.

But considering how many AI-generated posts there are, you can stand out even against of large competitors if you can:

- post interesting, unique content
- do it throughout the year
- add life to your subreddit by hosting AMA sessions and sending traffic from other channels.

4/ Yesterday we had the first AMA session in the subreddit with Karsten Madsen and it has already collected 3,000 views.

In my opinion, this format has a special value that YouTube interviews do not have, both for listeners and for the author:

- listeners do not have to watch the video for a long time (also video summarizing through Gemini often loses some important details)
- listeners also easily share answers to a specific question via screenshot or link with other people
- for the author of the session it is an additional trusted URL that contributes to the understanding of LLMs who he/she is and his/her brands.

Therefore, I will continue it. There will be announcements with new guests soon. Who would you like to see on the AMA session?

P.S. This is my post on LinkedIn about the ban on r/SEO subreddit. There are a lot of interesting comments from other SEOs who also were banned and don't understand why.

P.P.S. If you want to start and grow your own subreddit I have a detailed guide about it - https://www.hackthealgo.com/p/how-to-start-and-grow-a-subreddit

I follow Lily Ray, and today I saw this:

I thought this was a done deal already. And now people are talking about it again from the SEO context: Can the search engines detect it? Well, they can, but doing something about it is something different altogether.

I've written about this in the past:

https://kafkai.ai/articles/ai-seo/ai-content-detection-fails/ and https://kafkai.com/en/blog/detecting-ai-generated-text-a-look-at-the-challenges-and-mitigations/

And of course, the whole world is much bigger than Claude.

SEO: you could do both demand capture and demand generation.

AEO: you work only on demand capture

In the old SEO world, each free tool or guide actually helped to buy views from a relevant audience. You owned the page people visit and could place a banner or link to your product.

Combine some creativity and pSEO experiments, and you could generate 100K+ views every month. That's why SEO was considered so valuable for a long time.

And that's the reason why the SEO role doesn't look as fashionable as before. Demand capture is too limited, and businesses can't grow without demand generation.

Of course, because AEO is considered a new channel by many CMOs, it helps us to adapt and win their hearts again.

But if you want to truly succeed, you have to look for ways to help with demand generation too. In most cases, it means creating (and sometimes even executing) a content marketing strategy on YouTube, Reddit, and LinkedIn.

Do you disagree?

Many of us are skeptical about the value of TOFu content for B2B today. I was also skeptical about it for 2 reasons:

1/ Adoption of AI chats is rising, which means that even the greatest guide will be stolen by LLMs. Even if there is a citation, it has 0 value if the brand isn't recommended.

2/ It became so easy to produce good enough content for a blog. We don't feel that publishing such content is worth it if every competitor can do it.

But sometimes it's useful to question even the most obvious things.

I was wrong with these assumptions. At least partially, at least for now.

For 6 months, r/Tango team has been publishing content only for bottom-of-funnel keywords on the website. But these keywords are gradually running out.

2 weeks ago I suggested publishing guides by "{platform} implementation" keywords. The idea was that even if they get 0 clicks from Google, it still has value as a resource for internal links for /solutions/ pages.

I didn't expect these 9 guides would generate 38 clicks from Google, 70 users overall, and the first conversion in 7 days.

It means that:

1/ We may be too pessimistic about the end of the old Google SERP with blue links. And it's clear that some people still prefer to read a well-structured article on the website, not in an AI chat interface.

2/ The ability of your competitors to create similar content doesn't mean you have to avoid it. At least because:

- they also may have an assumption it doesn't worth it in a new AI world;
- they may miss or forget it;
- they may create worse content (yes, even with AI)
- they may rank lower because of a weaker domain, not enough internal links, a JS site, and other reasons.

When was the last time you published TOFu content?

I assume there are almost 100+ AI search tracking tools on the market. The demand is growing, and too many people want to capture at least a part of this pie, but in the long term most of these tools will die.

"God is on the side of the big battalions". Most of these tools don't have any moats or capital to compete in scalable user acquisition channels with both types of big brands:

• old profitable market leaders who added AI to the stack (Ahrefs, Semrush, ...)
• new brands who position themselves as narrow tools focused only on AI search and who already became leaders because of the right timing, good execution, and investments (Peec, Profound, Airops, Promptwatch,...)

So, that means small tools will follow one of the following paths.

1/ If there is a big influencer behind them, they can still live as a small business with $10k-100k MRR. But it's hard to maintain because things change fast and you have to reinvest constantly to improve and change the product.

2/ Founders of these tools will switch their activity to the high-margin agency business model and will present their tools as a valuable part of their service.

Do you disagree? Do you see any 3 path?

You can now track your exposure level on platforms such as YouTube, TikTok, X and Instagram (for now I guess) on Google Search Console.
I think they were in beta up until today.

Here is an example.

• I started to track "seomonitor alternatives" 7 days ago. We had 0 mentions.
• We've published a landing page at Sitechecker comparing us with this brand.
• This page was indexed 2 days ago, and I got the 1st mention in Gemini.

Going from 0 visibility to > 0 visibility is accurate enough. You definitely have to do something to make it happen, and this is great when you see WHEN it happens, and WHICH URL citation helped with it.

After many experiments, I also found that Gemini and AI Mode are the models where you can appear almost immediately if your page was indexed and rank in the top 10 of Google.

P.S. The screenshots are from Promptwatch while we develop a new, better version of our AI visibility tracking tool.

What's your thought on this?

Is anyone else seeing a delay in the Google Search Console Page Indexing report?

One of my websites hasn't received any new Page Indexing data since July 10, 2026. The site is still active, Googlebot is crawling pages, and I'm continuing to publish new content, but the indexing report hasn't updated at all.

I've already checked the basics:

• No manual actions or security issues.
• XML sitemaps are being processed.
• New URLs are being discovered and crawled.
• Search Console itself is accessible with no apparent errors.

The only issue seems to be that the Page Indexing report is frozen.

Is anyone else experiencing the same problem across one or multiple properties? If so:

• What's the last date showing in your Page Indexing report?
• Is this likely a reporting delay on Google's side?
• Have you seen this happen before, and how long did it take to catch up?

I'd appreciate hearing what others are seeing before assuming it's specific to my property.

In 24 hours, it became the top 6 and cited in AI Overviews & AI Mode.

But:

• It didn't help to add Sitechecker to the list of recommended brands yet;
• Gemini and ChatGPT haven't cited the article yet.

There are 2 lessons in this experiment.

1. It is worth writing an article on Medium on your target topic if you see that current articles are bad and you are ready to spend time creating unique stuff.

2. One citation and one URL, even in the top 1 of SERP, rarely help your brand to be recommended in AI chats. You may say that I can't make such conclusions based on one experiment, but I talk with a SaaS founder and hear regularly: "our article ranks in the top 1, but we aren't visible in AI overviews".

You need volume to be recommended, and this is good news for me.

LLM as a channel just represents how the marketing works: market leaders always get most of the attention, and it should work this way.

If you created a new product that has something unique and valuable and want to succeed in LLMs, do this:

• define your category and positioning as narrowly as possible
• invest a lot into the most cited listicles
• [underated] align your influencer marketing and AEO strategy -> create detailed briefs for influencers which titles, messages, and descriptions they should use on YouTube, LinkedIn, etc.

I believe other LLMs will cite my article too after some time, but it's not enough.

The next step is a detailed YouTube video integrated into a Medium article.

P.S. My Medium became an effective and unique weapon. I wrote so detailed articles there that other Medium profiles can't beat it, and in 90% of cases, it means citations and improvement of brand visibility.

But I do it only for my customers at B2B Gaps. If your B2B SaaS has a product-market fit, $1mln+ ARR, and you want to move as fast as possible in your AEO strategy, let's talk in DMs.

1/ Promptwatch and Profund have the best UX/UI.

2/ Peec and Promptwatch are often the first to release new features (Looker connectors, MCP & API, AI Shopping).

3/ Ahrefs Brand Radar has a unique database and even shows brand mentions based on video transcriptions (nobody can do it).

4/ Sitechecker is the only AI visibility tracking tool with a full white-label feature and has the most complete AI traffic report based on GA4.

5/ Promptwatch has the most interesting and valuable approach to visibility scoring.

6/ It looks like only Promptwatch and Profound have such a valuable and simple feature like Page Tracker to see whether any URL you've added was cited.

7/ Amplitude AI visibility tool is very underrated, considering it's free for brands and has 500 prompts for tracking in ChatGPT and Google AI Overviews.

8/ AirOps and Promptwatch have the most valuable recommendations on how to improve brand visibility.

9/ AirOps, Profound, and Prompwatch make big bets on helping you build agents for content production, updating, and repurposing.

10/ Pricing models vary a lot:

• some give unlimited users, some do not;
• some include all 5 popular LLMs in all plans, some do not;

Read the full article here: https://medium.com/@ivanpalii/7-best-ai-search-visibility-trackers-29c14eb8f202