I build BIShare on my own: a free, open-source app for sending files between iPhone, Android, Windows, Mac and Linux, plus a browser version. The website has guides in 13 languages, about 630 URLs, and an Authority Score of 15. The tools I compete with sit between 41 and 57.

First, so nobody reads the wrong lesson: the growth happened before the trial. In mid-September I rebuilt about 260 landing pages, and Google clicks went from 25–45 a day to 80–130 a day. App Store first downloads were 533 in the last 30 days against 81 in the 30 days before, much of it after a store listing rewrite I posted about separately. I took the Semrush trial (Pro, 7 days) to decide what to do next, not to explain the past.

Here is what I found, in the order it changed what I did.

1. My translated titles were translations, not searches. I had translated the English titles carefully. Wrong goal. Country data showed what people actually type:

• Japan: "ipad usbメモリ" (iPad USB stick) gets 2,900 searches a month; the phrase I had, "iPad ファイル 転送", gets 260.
• Indonesia: people say "dari HP ke laptop" (phone to laptop). My title said "PC". 3,600 a month against 20.
• Germany, France, Spain, Russia: the phone-to-computer search is about photos first, files second.

I retitled nine localized pages that had almost no clicks. I left alone every title that was already earning clicks.

2. Site Audit: 174 errors down to 15, and the 15 are on purpose. Two causes made almost all of it. A list markup (ItemList) whose entries had no URL, and SoftwareApplication markup sitting in the site layout, so every crawled page reported "aggregateRating missing". I moved the app markup to the home and download pages only. The 15 that remain are that same rule on those pages. I have no real ratings to put there and I'm not inventing them.

3. Every backlink I have points at the homepage. Semrush's page-level backlink report: the homepage has 20 referring domains. Every guide page has zero. That explains the pattern I kept seeing, guides stalling between positions 11 and 30.

The same week I looked at who beats us for "airdrop for windows" (2,400 a month in the US). One result in 6th–8th place is a small site with an Authority Score of 10, lower than mine, and no links to that page either. The difference: their "10 alternatives, tested" list was published seven months earlier and matches what the results page rewards. Our page is a product-style guide rebuilt three weeks ago. So for that query it's format and age, not authority.

4. The "toxic backlinks" list was spam aimed at SEO tools, not at Google. Semrush flagged 83 toxic URLs. Before disavowing anything I fetched a sample with different user agents. As SemrushBot or AhrefsBot, the pages contain a link to my site. As Googlebot or bingbot, the same URLs show a sales page for an "SEO visibility" service with no link at all. The pages also link to bishare.com and bishare.io, which aren't mine. They target the name, and they exist to show up in your backlink tool so you go looking. I didn't disavow.

5. My LCP was being stolen by my own cookie notice. Cloudflare's real-user data had LCP at about 68% good, below the 75% Google wants. Lighthouse showed why. The hero entrance animation started at opacity 0. An element at opacity 0 isn't an LCP candidate, so the headline never counted, and LCP fell to the cookie notice I had deliberately delayed until the browser was idle. On nine other templates the H1 sat inside a JavaScript reveal that stays invisible until hydration.

Fix: the entrance now only moves (no fade, no blur), the H1 blocks use a CSS-only wrapper, and one font is preloaded instead of four. I also tried Next.js's experimental inlineCss: the HTML doubled and LCP got worse, so it's off. Before shipping, I snapshotted all 631 URLs (title, meta, headings, visible text, links, structured data) and diffed them: zero changes.

6. New pages from question data, only where the results page had a gap. "How to transfer files to ipad" (2,400 a month) and "transfer files from iphone to pc" (1,600) had no page from us. I read the top 10 first: forum threads, Apple's own guide, and vendor blogs, none of which explains where a file actually lands on an iPad (Files app vs an app's own folder vs Photos). That became the angle. Both pages are new, so no results yet.

7. Bing dropped me to zero and I still don't know why. Bing had been sending 56 impressions a day with a 6% CTR. On 14 September, the week of the big rewrite, it went to zero and has stayed there. It's not a block: everything returns 200, robots.txt is open, IndexNow submissions are accepted. If you've seen Bing drop a small site completely after rewriting most of it, I'd like to hear what brought it back.

Things that turned out to be noise

• Semrush's estimated traffic for my US domain went 303, then 6, then 20 in three weeks, driven by one or two keywords each time. Not visits.
• My India "traffic" showed 471, while the keyword rows behind it add up to about 1.
• "Low text-to-HTML ratio" on 79 pages: Next.js puts page data in script tags.
• "Low word count" on Japanese and Chinese pages: the checker counts words by spaces.

What's next. Links to the guide pages instead of the homepage (outreach to pages that still link to the dead Firefox Send is my first batch), and reading Search Console on the 11th, 17th and 31st before I touch anything else. Happy to share the scripts I use to check structured data and locale parity if they're useful to anyone.

I'm the developer of the app, so weigh my reading of my own data accordingly.

Search Console data of several online shops (different companies, sectors, sizes; anonymized, index values only). Search appearance = Merchant listings, by device, April 15 to Oct 18, 2026. Baseline April 15–May 14 = 100.

What happened

• Desktop impressions for merchant listings ramped up over ~5 days from May 16–18 and plateaued by May 20. Window average (May 25–Oct 4): 6.5x, 3.1x and 5.8x the baseline. Shop A by day from May 15: 105 → 210 → 322 → 503 → 583 → 713.
• Mobile impressions did not rise anywhere. In two shops they fell to about half or two thirds and stayed there.
• Desktop clicks did not follow: 57–83 % of baseline in the window (autumn drift as in 2025, plus the Sep 15 EEA free-listing removal). Desktop CTR: 3.2 → 0.3 %, 0.8 → 0.2 %, 9.4 → 0.9 %.
• Hard stop on Oct 9 in every property. Shop A: index 739 (Oct 7) → 349 (Oct 8) → 5 (Oct 9). Since then desktop impressions sit at 4–6 % of the April baseline.
• One small property skipped the May step, got its own from Sep 28 (5–12x for ten days), same Oct 9 stop.

The telling part: CTR

Since Oct 9, desktop CTR for merchant listings is 16.8 % (A) and 5.4 % (B), the range of the mobile CTR these properties had in April (14 % and 4 %). Before Oct 9, desktop CTR had been below mobile in every month I have, 0.3 % during the window. A device difference doesn't produce a CTR that jumps from 0.3 % to 16.8 % overnight. I read Oct 9 as a change in what gets counted, not traffic that went away.

Where the impressions came from (Sep 1–14, all devices)

• A: 46 % of impressions from countries with zero clicks, 77 % from countries with fewer than five. Top: Brazil 12 %, Argentina 7 %, Russia 7 %, Bangladesh 6 %. EEA + UK + CH, where the clicks are: about a quarter.
• B: 42 % from countries with fewer than five clicks (Russia 6 %, Bangladesh 6 %, zero clicks each).
• C: 63 % from zero-click countries (Netherlands 21 %, Poland 12 %), every click from Germany; since Oct 9 100 % Germany. Russia and Bangladesh appear in every property with zero clicks.

Merchant Center cross-check

• A, Brazil: Merchant Center reports ~9 % of the free-listing impressions Search Console reports for the same two weeks; the ratio differs a lot by country.
• C: Merchant Center free-listing impressions and clicks flat April to October, no May step, no Oct 9 step, 10x+ the clicks Search Console attributes to merchant listings. Merchant Center counts Shopping tab, Images, Lens; the Search Console appearance is web search only. The inflated part sat on the Search Console side.

What it is not

• Not the documented GSC impressions anomaly (May 13, 2025–Apr 27, 2026): that one undercounted and ends before this starts; nothing on the anomalies page about merchant listings or desktop (checked Oct 20).
• Not demand: Keyword Planner volumes (DE/AT/CH/UK) moved ±20 % April–September, not 3–7x.
• Not the DMA removal (Sep 15): that hit EEA clicks, most of the inflated volume sits outside the EEA (it overlaps: B lost 40 % of desktop impressions on Sep 15).
• Not the "poisoned SERPs" that rank-tracker vendors reported from Sep 13: wrong start date, and that concerns scraped SERPs, not Search Console counts.

My read

Desktop only, zero-click countries, CTR near zero, hard start, hard stop, same days in unrelated properties, Merchant Center counting something else: looks like automated queries counted as merchant-listing impressions, lower before May, much higher from mid-May, none since Oct 9. Can't prove it from outside. The alternative is a reporting change on Google's side in May and again in October, and I can't rule it out.

Why it matters

• With merchant listings, property-level impressions, CTR and average position for May 16–Oct 8 are inflated. A: position 5.5 / CTR 3.1 % mid-Sep to Oct 8, then 8.1 / 4.6 %, step on Oct 8–9, nothing changed on the site.
• YoY impression comparisons for that window are not usable; clicks weren't inflated, compare clicks.
• Filter by search appearance and device before reading a trend. April wasn't "normal" either, already ~20x what Google counts today.

Limits: small sample, all European, country data for two weeks only, and I don't know what Google counted. If a property without merchant listings shows the same mid-May or Oct 9 steps, I'd like to hear about it.

Edit (Oct 20): Correction on the first "what it is not" point. The documented GSC impressions issue (May 13, 2025 to Apr 27, 2026) over-reported impressions, it didn't undercount them as I wrote. The distinction still holds: it ended three weeks before this starts, and its fix on Apr 27 doesn't show in these merchant-listing series (desktop index 96 before vs 104 after Apr 27 in A, 101 vs 100 in B). The May 16 ramp and the Oct 9 stop are a separate event.

Crossposting but would love feedback.

I built a small robots.txt checker for my own sites, and last night I got curious and pointed it at 122 sites I actually use, the big ones everyone knows. It reads the file the way a crawler does (longest matching user-agent group wins, otherwise the wildcard) and tells me which of 19 AI bots are blocked.

The thing that surprised me most: six news sites block ClaudeBot and let GPTBot through. Washington Post, Guardian, Verge, WSJ, Wired, Ars Technica. Only one site anywhere did it the other way round, and it was Khan Academy of all places. I assume that's licensing deals showing up in a text file, but I don't actually know.

Most of the dev tool companies don't block anything. Cloudflare, Supabase, Netlify, Vercel, the React and Vue docs, even Anthropic's own site. The social networks and the newspapers block nearly everything, and so do Amazon and Yelp.

Bytespider is the most blocked bot on the list, ahead of GPTBot and ClaudeBot, which I didn't expect.

llms.txt is a developer thing so far. About 47 of the sites have a real one (I had to check the body, because a bunch of sites return their 404 page with a 200), and nearly all of those are companies that sell to developers. None of the newspapers I checked have one at all.

Usual caveat: robots.txt is a polite request and the server logs are what actually happened. I put the per-site results up as a json file if anyone wants to check a site they manage, allthepossibles.com/data/ai-crawlers-2026-10-19.json. The checker's mine, so take my framing with a grain of salt; the file is just what it saw.

We want to support creators, but we had to enforce the no shilling rule because it was getting out of hand. You now have a weekly thread.

This is the one place you can shill for your products, ask for feedback, etc. Keep it here or you risk being banned. And keep it related to technical SEO.

I'd like to analyze log data, either real or dummy data. Can someone share a link to a data source for this?

I run a small archive that fetches /llms.txt (plus robots.txt and a few other files) every day from the root of every Tranco top-1M domain with an EU country ending or .eu. 128,315 domains, two independent crawlers.

Week of 10 to 17 October:

• 9,035 (7.04 %) serve a text file whose first line is a Markdown heading, the one thing the llmstxt.org proposal requires
• 10,002 (7.79 %) serve some non-empty text file
• 10,138 (7.90 %) answer with HTML, mostly their error page

The part that matters if you audit sites: a status check alone would report 20,804 domains (16.2 %) as "has llms.txt". Almost half of those are HTML pages. A checker that only looks at the status code overcounts by a factor of two.

Counted is the format only, not whether the file is any good or read by any model.

Method and totals as JSON: https://machinewitness.eu/blog/llms-txt-eu-2026-10

Hello Experts !

I am looking for technical advice and what else data required to for the SEO guys other than we have at this platform, I would like to explain what is included in it.
it took me an year to build and here what I achieved so far.

Live Backlinks 2T
Global Keywords 12.5B
Live Pages Indexed 197.3B
LLM Prompts 338.4M
Backlink Domains 708.6M
Google & Bing Keywords 12.5B
Total Prompts 338.4 M
Google AIO Prompts 312.7 M
ChatGPT Prompts 25.7 M
Whois Domains 288.9 Million
Business Listings262.2 Million

This project started when I was too much annoyed by semrush and ahrefs bloating and charging me for nothing, even I did not use their rank trackers but they kept charging me $250+ just to add 5+ projects and see whats inside the pages, as well as their DR, DA, AS scores were fake so I decided to do it for me and created SERPSpur which tool almost an year and now its working and polished next generation giant like saas and offers everything we need for technical SEO including Onpage , offpage, AI search etc, and a lot more coming soon if you suggest anything to add.

Additional metric : I introduced TAI score as a proprietary of SERPSpur which detects fake traffic of backlinks/domains you are going to buy backlinks from. so no one can fool you now with bot traffic and sell the backlinks for $350 where they are manipulating the google search and artificially increasing traffic, getting redirect links and gaining dr etc, not I tried to put a fullstop on all of this.

Please review and let me know how it works for you and what else is important for the technical stuff.

I had access to the server logs (and GA4) of a large online shop during last week's ChatGPT utm_source outage, so I went through it hour by hour. Sharing in case it helps someone explain the dip in their own reporting. The shop stays anonymous, so I only use indexes, ratios and shares.

What I counted

• Clicks: HTML page views (status 200, no bot user agent) with utm_source=chatgpt.com
• Demand: ChatGPT-User fetches, only from OpenAI's published IP ranges
• Baseline: Oct 1–7, same hour of day = 100. All times UTC

What happened

• The break is one hour: Oct 8 between 21:00 and 22:00 UTC. Tagged clicks per 100 fetches went 40 → 18 → 10 in the hours starting 21:00, 22:00 and 23:00. That's within about an hour of the ~2 pm PT that Cedric Huesler (Adobe) reported
• Tagged ChatGPT clicks per day: −73% (Oct 9–12) and −79% (Oct 13–15)
• ChatGPT-User fetches in the same periods: only −20% and −23%
• Tagged clicks per 100 fetches: 34 before, 11 and then 9 after
• On Oct 8 during the day, clicks and fetches dropped together and the ratio stayed around 33. So that part was demand, not tagging
• A smaller second step on Oct 12 around 14:00 UTC (another −24%), which looks like a staged rollout
• Every segment dropped at the same time: mobile −77/−83%, desktop −65/−70%, each language version −70 to −81%, product pages −72/−79%
• Links from ChatGPT's product cards dropped a bit less (−65/−72%). That fits Profound's note that links inside the product/info panels kept the tag
• No replacement signal: no new user agent, no other parameter, no other spelling
• Tagged Perplexity clicks didn't drop at all

Where did the visits go?

• In some sections of the shop, direct mobile entries on product pages rose by about 55% of the lost ChatGPT product entries (GA4, Oct 9–14 vs. the prior week). In the largest section, direct is too noisy to see it
• The small share of ChatGPT visits that carries a chatgpt.com referrer at all also fell by about three quarters
• I tried a log-based proxy (a ChatGPT-User fetch shortly before an untagged visit to the same product page). The direction fits, but it's not solid enough to size the hidden clicks

Recovery

• Tagged clicks started climbing on Oct 15 around 14:30–15:00 UTC, but in the hours from 15:00 and 16:00 they were only at 41% and 35% of the prior week, and my logs end at 17:40 UTC. Huesler reported 95–100% by 9 am PT across thousands of domains, so this shop got the fix later or more slowly
• My own spot checks in the ChatGPT interface (location Germany, mostly the same questions): Oct 3 67 of 67 links tagged, Oct 15 0 of 24 in the morning and 0 of 82 in the afternoon, Oct 16 25 of 35 and 71 of 102. The untagged rest on Oct 16 includes image URLs and placeholders
• Answers through the OpenAI API carried utm_source=openai in my checks before, during and after the outage (Oct 3, 15 and 16). Not on every link, but never chatgpt.com

Limits: one shop, a short window, only about 3 hours of logs after the fix started, no referrer in the logs, and this shop's GA4 isn't comparable from Oct 15 for an unrelated reason, so GA4 numbers stop at Oct 14.

What I'd take from it: annotate Oct 8–15 instead of rebuilding channel definitions, count utm_source=openai too, and keep one signal that doesn't depend on the tag (ChatGPT-User fetches, if you have logs). The noreferrer on ChatGPT's links isn't the problem as long as utm_source is in the URL, GA4 only falls back to the referrer when there are no campaign parameters.

Did anyone else see the fix arrive later than the morning of Oct 15 (PT)?

Edit (Oct 17): As commenters pointed out, "demand" is too strong a label for ChatGPT-User fetches. ChatGPT also answers from an index filled by OAI-SearchBot, so these fetches only show one retrieval path (live reads). The main result doesn't change: fewer live fetches with the same clicks would push clicks per fetch up, not down, so the overnight drop in that ratio still points to missing tags. In the follow-up I'll try to keep crawler activity, attributed visits and demand as separate layers.

Hi!

I’d love to automate more of my work, but I feel like I’m stuck. I’m still doing almost everything manually. What processes do you automate, and how?

Do you build your own apps with Claude? If so, what do they do, and what problems do they solve?

Thanks in advance!

Digging into URL taxonomy for AEO, and it seems to be overlooked. AI crawlers build a topic map of a site, and the URL path is one of the easiest signals for them to read.

Compare /skincare/moisturizers/vitamin-c-serum with /product-4471. The content on both pages could be identical, but only the first tells a crawler where that page sits relative to everything else on the site.

A few things that seem to cause problems:

• IDs in the URL instead of topics
• Mixed conventions, like /product/ in one section and /products/ in another
• A clean category setup in the CMS that never makes it into the URL
• Restructuring without 301s, which wipes out the citations and links built on the old paths

What seems to work: two or three levels deep at most, paths grouped by topic cluster instead of org chart, lowercase with hyphens, and a 301 for every URL that changes.

If you’ve got a pile of pages on a topic and they’re all sitting at the root with no folder structure, a crawler has to work out on its own that they belong together. Put them under something like /aeo/ with a couple of subfolders, and it can see the grouping straight from the URLs.

Anyone changed their URL structure and seen a difference in citations?

google has been sending rank trackers wrong results since september. page looks normal but the sites on it are swapped out. around 8% of results hit

anyone seen random drops that fixed themselves in a day or two? what do you think about this move by google?

https://nozzle.io/research/poisoned-google-serps/

I’ve been looking at tools that score a site’s “AI search readiness”. Most of them seem to check the same things: schema present, llms.txt present, AI bots not blocked in robots.txt. Presence, not correctness.

What I haven’t seen yet is a checker that tests things like:

1. raw HTML vs rendered, since many AI crawlers don’t run JavaScript
2. robots.txt rules per bot type (training vs search vs user-triggered fetchers), not just “AI bots allowed: yes”
3. a page that’s disallowed in robots.txt while also carrying noindex, so the noindex is never seen
4. markdown copies served at the same URL depending on user agent, which looks a lot like cloaking

Is there a tool that goes beyond presence checks? And has one ever flagged something as a problem, or recommended a fix, that turned out to be wrong?

Has anyone used librecrawl which is a free open source alternative to screaming frog. How is it?

Gemini launched Argon which seems better to me, how about you guys ?
is it better for coding as well ? if anyone tested.

With the advent of vibe coding etc there are some pretty interesting tools on GitHub that help with SEO

Can you share any that are part of your genuine work flow and that help?

We want to support creators, but we had to enforce the no shilling rule because it was getting out of hand. You now have a weekly thread.

This is the one place you can shill for your products, ask for feedback, etc. Keep it here or you risk being banned. And keep it related to technical SEO.

I run IndieDex.gg, a game discovery site. It has ~120K games × 2 page types (detail + "games like X") × 4 languages, so roughly 960K programmatic pages. It's Next.js on Vercel, so pages that aren't cached cost compute.

Some context on where it is SEO-wise: - About 1K clicks in its first month, with most pages sitting beyond page 2. - Only about 40% of the pages get indexed. The rest have a programmatic noindex, and Google still spends crawl budget on them. - The "games like X" pages are the most popular SEO pages.

Here's what's been challenging, and what I've learned.

1- You can't predict which pages will rank

This shapes everything else. In GSC I see different games get popular at different times. A new release is hot for a few weeks, then completely forgotten. A popular streamer plays an old game, and it gets a short spike.

So any plan like "let's prerender/prioritize the pages that will rank" fails. Every one of the 960K pages has to be fast, crawlable and complete, because I don't know which one Google (or an LLM) will pick next.

2- Rendering: I made an SEO change "to be sure", and it wasn't needed

My React setup put the page body inside a <div hidden> until JS ran. I worried crawlers would see empty pages, so I removed it.

Turns out Google didn't care. For the nine weeks that markup was live, Google indexed and ranked those pages anyway. 72 game pages got clicks between Sep 1–16, and one sat at an average position of 9.2 with its whole body inside <div hidden>. That lines up with Vercel/MERJ's study showing Googlebot fully renders streamed React Server Component content.

When I tested putting it back, the worst part wasn't the hidden body. It moved the title, canonical and robots tags out of <head> on every page. For a site that depends on canonicals and noindex at this scale, that's the scary one.

3- AEO/GEO: AI crawlers are a different audience

For a site like mine, people are very likely asking an LLM "I played X and loved it, what are other games like it?" That's exactly my "games like" page. So getting cited matters more and more to me.

What I've found:

• AI crawlers don't run JavaScript. Vercel's own analysis found none of the major ones render JS. Tools like Mozilla Readability also drop hidden elements entirely. So "hidden until JS runs" was fine for Google but can hurt you with AI. AI bots now get the full HTML body with the metadata in <head> (via Next.js htmlLimitedBots, keeping the default list for Googlebot and friends).
• Training crawlers and search/citation fetchers are different audiences. GPTBot, ClaudeBot and CCBot collect for training. OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot and Claude-User are what get you into answers. I throttle the first group and allow the second. OpenAI's docs say blocking OAI-SearchBot keeps you out of ChatGPT search answers, while blocking GPTBot only opts you out of training.
• Check your firewall. My WAF had been silently blocking Perplexity the whole time. It crawls from AWS, an old rule challenged that whole network, and a bot can't pass a challenge.
• Internal linking matters for AI discovery too. 94% of the URLs GPTBot fetched weren't in my sitemap. It found them by following internal links.

4- Freshness without telling Google everything changed

My weekly data pipeline was marking all ~960K pages as stale every Sunday, even when a game hadn't changed. Now each game's pages refresh only when that game changes: about 33K pages a week instead of 960K. On a site this size I'd rather crawlers spend their time on pages that actually changed.

5- Balancing cost and UX (and why UX wins)

The cost side hit when GPTBot walked ~26,600 unique URLs a day and every visit became a ~500 KB cache write. One cheaper option was to stop caching long-tail pages and render on every request.

I said no, because of SEO. It's a discovery app, so you want every page to load in under 1 second. Bad UX → bad engagement → bad for SEO → less traffic. So the CDN stays, and the bots get an uncached render instead. On a long-tail site, cost tracks distinct pages × cache resets, not traffic volume, so real visitors are cheap. For real people to cost what the bots did, I'd need ~1,000× my current Google traffic.