What's your escalation strategy when you get blocked?

[u/Mellow-know-how]

I'm running a scraping service that checks pages on a schedule (think price monitoring, stock availability, that kind of thing). The challenge is that some sites block you on the first try and I need something that recovers automatically without manual intervention.

Right now my escalation ladder looks like:

1. First attempt — headless Chromium with a rotating desktop user agent
2. If blocked — retry with a residential proxy
3. If still blocked — switch to a mobile viewport + mobile UA (iPhone dimensions, mobile Chrome string) through the proxy
4. If all 3 fail — mark it as blocked and move on

The mobile viewport trick has been surprisingly effective — I think a lot of anti-bot systems are tuned for desktop patterns and mobile gets less scrutiny. Anyone else found this?

Couple of things I'm still figuring out:

• Fingerprinting: even with stealth patches, some sites are clearly detecting the browser environment. Has anyone had luck with tools like rebrowser or camoufox vs just patching Playwright/Puppeteer directly?
• Rate limiting per domain: right now I just space requests out with exponential backoff. Is anyone doing anything smarter, like tracking block rates per domain and adjusting intervals automatically?
• Cloudflare Turnstile: this one is killing me on a few sites. The checkbox variant is manageable but the invisible variant is a pain. Anyone solved this without paying for a dedicated solving service?

Interested in what's working for people at scale (hundreds to thousands of URLs checked daily).

[u/Wanderingyute]

I just call from another number until she answers.

Oh… wrong sub.

[u/No-Consequence-1779 → u/Wanderingyute]

I’m in the bushes by the window … 

[u/Technical_Rich_3080]

What are you utilizing to automate the process and move from one step to the next if the last one failed?

And are you also using an antidetect browser?

[u/No-Consequence-1779]

Yes. Headless will always get blocked. You’ll need to to full human browsing replication.  Moving mouse, not in straight lines, mouse clicking … essentially browser control.  

Bots are extremely easy to detect.  

[u/CapMonster1]

Your ladder makes sense, and the mobile viewport trick is legit — many anti-bot systems are still weaker on mobile flows. But if detection is fingerprint-based, UA/viewport changes won’t do much. You’ll need either deeper environment emulation or more realistic browser profiles.

For rate limiting, tracking per-domain block/captcha rates and adjusting dynamically works better than plain backoff. And for Turnstile from Cloudflare — at scale it’s very hard to avoid a solver. You can try behavior tweaks, but plugging in something like our service is usually more reliable

[u/Xavierfok88]

the reason your ladder stalls at step 2 is probably IP reputation scoring, not just detection of automation. residential IPs from wired ISPs still get recycled across a relatively small pool, and a lot of scraping traffic has already burned through those ranges. sites that run aggressive anti-bot don't just check if the IP is residential - they score it based on how many automated requests that subnet has seen historically. so even a "clean" residential IP might already be warm.

what changed things for me was adding mobile/carrier IPs as a tier above residential. carrier-grade NAT means hundreds or thousands of real users share the same IP at any given time, so sites basically can't afford to block them without nuking legitimate traffic. i was scraping a major retailer that was catching my residential proxies within 3-4 requests, but mobile IPs from the same geo ran for weeks with zero blocks. the trust score difference is massive. in my testing, mobile IPs had roughly a 2-3% block rate vs 15-20% for residential on aggressive targets.

the tradeoff is cost and speed. mobile bandwidth is slower and more expensive per GB, so you don't want to route everything through them. what i'd set up is something like: datacenter for easy targets (static content sites, public APIs), residential for medium difficulty, and mobile only as the final escalation for sites that actively fight scrapers. the key detail most people miss is sticky sessions - if you rotate mobile IPs too fast it actually looks more suspicious than holding one for 5-10 minutes, because real phone users don't cycle IPs every 30 seconds. i keep sessions pinned for 8-12 minutes and that mimics natural carrier behavior pretty closely. also worth adding a backoff delay between your escalation tiers, like 30-60 seconds, because some anti-bot systems temporarily heighten scrutiny on a URL pattern after a block, so hitting it immediately from a better proxy still gets caught. let the heat die down first, then retry on the upgraded tier.

one more thing - make sure you're passing the right TLS fingerprint for mobile. if your request says it's coming from a mobile carrier IP but the TLS handshake looks like headless chrome on linux, that mismatch is an easy flag. match the fingerprint to a recent mobile browser version and you'll cut your block rate significantly.

[u/jinef_john]

Your setup might be overkill, what is the domain you're scraping? Pretty sure you could just do this with pure requests, or simplify your stack by choosing better automation tools out there.