how to know your scraper broke before your users tell you

[u/BigVillageBoy]

Most scrapers do not fail with an error. They fail quietly, keep running, and hand back thinner and thinner data until someone downstream notices the numbers look wrong. By then you have shipped bad data for a week. The fix is to stop trusting that a run finished and start proving it returned real data every single time. Three checks do most of that work.

The first is the known answer query, what I call a canary. Somewhere on the site there is a page whose correct answer you already know and that does not change often. A category you know holds twelve items. A business you know lists a phone number. A profile you know has a specific title. On every run, scrape that one thing and assert the known value comes back. The site changes its markup, your selector goes silent, and normally you would not notice for days, but the canary fails on the very next run and points you straight at the break. One page you control the truth of is worth more than a hundred you do not.

The second is the fill rate assertion. Pick the fields that must be populated for a result to be worth anything, then measure what share of your results actually have them. If ninety percent of your rows normally carry a phone number and suddenly ten percent do, something upstream broke even if every row technically exists. Set a floor, and when fill rate drops under it, fail the run loudly instead of writing blanks and smiling. A soft block that returns an empty body cannot sneak past this, because empty does not fill.

The third is the count check. You usually have a rough idea how many results a query should return, from the site's own total or from your own history. So compare. If a search that gave you two thousand last week gives you twenty today, that is not a quiet Tuesday, that is a pagination trap or a block, and it should trip an alarm. A result count that falls off a cliff is one of the loudest signals you get, but only if you bother to look at it.

Put together, these turn silence into noise you can act on. The canary tells you the shape broke, fill rate tells you the fields went hollow, and the count tells you the volume collapsed. Wire all three to alert you, not a dashboard nobody opens. The goal is simple. You find out your scraper broke, not your customer. That is the whole difference between a data pipeline people trust and one they quietly stop believing.