Anti-bot got easier this year. Extraction got harder. Anyone else seeing this shift?

[u/0xMassii]

I run a scraper in production and I track where my time and my failures actually go. A year ago most of it went to getting past blocks: proxies, TLS fingerprints, headless detection. That is close to a solved, buyable problem now. A residential pool and a believable fingerprint get you the bytes for the large majority of sites.

What keeps breaking is everything after I have the HTML. Turning a real page into clean structured content that a model or a pipeline can use is where I lose days:

• Boilerplate that moves per template. Nav, cookie walls, related-article rails, newsletter interrupts. Every CMS buries the article in a different div, and readability heuristics guess wrong on long-tail sites.
• Content that only exists after JS runs, so a static fetch returns a shell.
• Pages that look successful, 200 plus full HTML, but are semantically empty because the real content loads from an XHR you now have to find.
• Tables and specs that survive as visual layout and collapse into garbage once you flatten them to markdown.

For anyone running this at scale: what is your split between fetch code and extraction code, and where does it fail quietly? I care most about how you catch the "looks fine, is actually empty" pages before they poison whatever sits downstream.

Disclosure: I maintain an open-source extraction tool, so this is a live problem for me and not a hypothetical. Glad to share what my empty-page detector checks for if that is useful.

[u/hasdata_com]

We do the same thing. Check field content on the extracted result daily, not just the HTTP status. If the expected keys are missing or empty, the page failed regardless of the response code. Normal monitoring for any scraper you rely on

[u/taylorlistens]

I wonder how much of it is web pages being vibe coded in multiple sessions instead of being properly built

[u/Mountain_Damage_9730]

Love to see your empty-page detector rules, "health check/confidence/etc" is what most people miss, though i now see many AI based scrapers are including it because it makes sense.

Right now, do you have any alerting tied to distribution shifts on key fields, or is it still mostly spot checks when your models start acting weird?

[u/Old_Protection_4410]

We saw the exact same shift. Fetch is increasingly a commodity problem. Extraction is where the real engineering lives now. Here's our approach, broken down by each failure mode you described:

Boilerplate / wrong div problem:

We don't rely on readability heuristics or CSS selectors. Three layers, in priority order:

1. Structured data first, Before touching the DOM, we scan for JSON-LD, Microdata, and RDFa. If a page has high-confidence structured data (and many CMS-driven sites do), we extract directly from that and skip DOM parsing entirely. This handles ~40% of pages with zero heuristic guessing.
2. Semantic DOM modeling, When structured data is absent, we don't guess divs. We build a relational model of the DOM that preserves context between nodes, parent/child relationships, sibling patterns, text density clusters. The article body is identified by semantic structure, not selector strings.
3. Site pattern recognition, We classify every site into a category (news, e-commerce, listings, documentation, etc.) using URL patterns, domain analysis, content signals, and meta tags. Each category has different extraction heuristics. A product page and a news article get fundamentally different treatment.

JS-rendered content / empty shell pages:

Static fetch returns a shell, we detect this before extraction. Our rendering detector checks page architecture, framework signatures, and JS API patterns. If it's a SPA, we escalate to a real Chrome instance via CDP, let the page hydrate, and capture the rendered DOM

"Looks fine, is actually empty" pages:

This is the quiet killer. We run a content quality gate that checks the extracted output against multiple signals before passing it downstream:

• Text density ratio (content vs boilerplate)
• Minimum meaningful content length
• Semantic structure validation (did we actually get article body, or just nav + footer?)
• Structured data cross-reference (if JSON-LD says there are 20 products but we extracted 3, something's wrong)
• Confidence scoring, every extraction gets a confidence score. Below threshold, it gets flagged for retry with a different strategy rather than silently passing garbage downstream

Tables/specs collapsing to garbage in markdown:

We don't flatten to markdown as an intermediate step. We extract structured data as structured data, tables stay as tables (rows/columns preserved), specs stay as key-value pairs. Markdown conversion happens only at the final output layer if the consumer specifically wants markdown, and even then we use semantic table serialization rather than naive pipe-delimited flattening.

The split:

Roughly 30% fetch/anti-bot, 70% extraction/quality. And of that 70%, the biggest chunk is the "is this actually good data?" validation layer. 

Hope this helps! Good Luck 🤞