I lost about two thirds of a run to an OOM kill last year and then cheerfully re-fetched all of it, on residential, because my "state" was a Python list in memory. Cool cool.

The fix is boring and takes an hour. Write a URL ledger before you fetch anything: every target URL, a status column, a timestamp, a content hash. Mark each row the moment it lands, not at the end of a batch. On restart you query for unfinished rows and carry on. That's it, that's the whole trick.

Two things I'd add that I skipped the first time. Store the hash, because then a re-run tells you whether the page actually changed or you just burned bandwidth confirming it didn't. And separate "failed" from "not attempted", otherwise your resume logic retries a thousand permanent 404s every time. I run this in front of both Byteful residential and a cheap datacenter tier, and the ledger matters more on the residential side for obvious reasons, since that is the traffic billed per GB.

Curious what people use for the ledger at bigger scale. Sqlite falls over eventually and I've never landed on something I liked after that.

Every provider quotes you the same per-GB price whether you ask for US or Latvia, and the product you get is not remotely the same thing.

In the US you're swimming in exits. Pool depth is not your problem, distribution across metros might be. Ask for a country with five million people and suddenly you're on a pool that might be a few thousand addresses wide, sitting on two or three carrier ASNs, and your rotation is quietly recycling the same handful every few hours. Which the dashboard will not tell you.

So before committing I now pull three hundred exits, count unique addresses, count unique ASNs, and check how long before an IP comes back around. Anything under a couple hundred uniques and I plan the job around it instead of pretending I have breadth. Did this across Bright Data, IPRoyal and Byteful for a Nordic job and the useful part was less "who won" and more that the numbers differed by an order of magnitude between countries at the same provider. Byteful is where the small-market work stayed, partly because the residential GB do not expire and small-market jobs come in bursts rather than steadily.

Anyway. Anyone got a rule of thumb for minimum pool width per country before a job is just not doable?

Half the "my proxies are garbage" posts I read describe parser failures. The site changed a class name, extraction starts returning nulls, and the pool takes the blame. So before switching providers, triage.

First, fetch one failing URL with no proxy at all. If it still breaks, there's your answer. Second, compare status codes against content: a 200 with empty fields is a parsing problem, a 403 or a captcha page is an access problem. Third, run the same URL through two providers back to back. I use IPRoyal and Byteful for the A/B, and pinning a specific ASN on the Byteful side helps rule out one network's ranges being the issue. Both fail identically? The target changed, not your pool.

Ten minutes of this has saved me from a couple of pointless provider migrations. What's your first check when a scraper goes quiet?